One or more embodiments provides method for transmitting a signal by a transmitting apparatus in wireless a communication system supporting a modulation scheme based on a quadrature amplitude modulation (QAM) scheme and a frequency shift keying (FSK) scheme. The method includes determining at least one first bit which will be mapped to at least one QAM symbol and at least one second bit which will be mapped to at least one FSK symbol among bits included in an encoded input bit stream; and generating a modulation symbol by mapping the at least one first bit to the at least one QAM symbol and mapping the at least one second bit to the at least one FSK symbol based on a Hamming distance of the at least one second bit.