An apparatus and method are provided for a communication scheme for converging a 5G communication system for supporting a higher data transfer rate than a post-4G system with the IoT technology. A method includes configuring a loopback path between a transmitter and a receiver; transmitting a preconfigured signal from the transmitter to the receiver through the loopback path; identifying an image signal, wherein the image signal includes a distorted signal of the preconfigured signal; determining a gain error value and a phase error value based on the preconfigured signal and the image signal; and performing I/Q calibration based on the gain error value and the phase error value. A subcarrier in which the preconfigured signal is transmitted and a subcarrier in which the image signal is generated do not overlap.