A method and apparatus for performing demapping in a wireless communication system utilizing a modulo operation are disclosed. The demapping method of a receiver in a wireless communication system includes receiving an input signal and first information indicating whether a first modulo operation is performed on the input signal from a transmitter; if the first information indicates execution of the first modulo operation, performing a second modulo operation of the input signal, and acquiring a reception signal; generating a maximum function value having a highest probability that the reception signal corresponds to a candidate constellation point of an extended constellation; and generating a log-likelihood ratio (LLR) using the generated maximum function value.