The invention discloses a self-interference eliminating method in a two way full-duplex MIMO relay communication system. The method aims at addressing the problem of neglecting system capacity property while taking into consideration of minimizing self-interference of prior art. The method includes the following steps: 1. estimating channel parameters; 2. designing the transmission of a first source node, receiving a beam forming matrix to eliminate the self-interference of the first source node; 3. designing the transmission of a second source node, receiving a beam forming matrix to eliminate the self-interference of the second source node; 4. and designing the transmission of a relay node, receiving a beam forming matrix to eliminate the self-interference of the relay node, and at the same time selecting the transmission that conform to conditions, and receiving a beam forming matrix and maximizing the system capacity. According to the invention, the method eliminates the self-interference of the relay node as well as the self-interference of the two source nodes, can maximize the system capacity while eliminating self-interference, and can be used in distributed cooperation transmission system of future fifth generation of wireless mobile communication.