The present disclosure relates to a pre-5th-Generation (5G) or 5G communication system to be provided for supporting higher data rates beyond 4th-Generation (4G) communication system such as long-term evolution (LTE). An operating method of a reception device in a wireless communication system includes: receiving a signal transmitted through a plurality of antennas of a transmission device; determining an initial symbol vector based on the signal; determining a first candidate symbol vector based on a plurality of solution vectors which are obtained by a search on the initial symbol vector; determining a second candidate symbol vector by flipping at least one symbol value of the first candidate symbol vector; and determining a symbol vector transmitted from the transmission device based on at least a portion of the second candidate symbol vector.