An apparatus and method for reduces an overhead caused by an operation transition gap in a Relay Station (RS) of a relay wireless communication system. The method includes identifying a signal delay time with an upper node. A transmission/reception operation transition time is determined through a negotiation with the upper node. An idle time is identified. An overhead caused by a transmission/reception operation transition is determined with consideration of a start time point of an Uplink (UL) subframe dependent on the signal delay time, the transmission/reception operation transition time, and the idle time. And communication is performed considering the overhead. The start time point of the UL subframe is set earlier than a start time point of a UL subframe of the upper node in consideration of the idle time.