The present disclosure relates to a 5G or pre-5G communication system for supporting a higher data transmission rate successive to a 4G communication system such as LTE. The present disclosure provides a communication method of a Base Station (BS) using an Integer Forcing (IF) scheme in a Multi-User Multiple-Input and Multiple-Output (MU-MIMO) communication system, the method including: receiving a reception signal including a desired signal transmitted from at least one User Equipment (UE) served by the BS and an interference signal transmitted from at least one UE served by a neighboring BS; determining an IF filter considering the interference signal based on information on the interference signal received from the neighboring BS; filtering the reception signal using the determined IF filter; and detecting or decoding at least one of the desired signal and the interference signal using the filtered reception signal.