The present disclosure relates to a 5th generation (5G) or pre-5G communication system for supporting a higher data transfer rate beyond a 4th generation (4G) communication system, such as long term evolution (LTE). An operation method of a reception device in a wireless communication system comprises the steps of: receiving a signal transmitted through a plurality of antennas of a transmission device; determining an initial symbol vector on the basis of the signal; determining a first candidate symbol vector, on the basis of a plurality of solution vectors obtained by searching for the initial symbol vector; determining a second candidate symbol vector by changing at least one symbol value of the first candidate symbol vector; and determining a symbol vector transmitted from the transmission device, on the basis of at least a part of the second candidate symbol vector.