A method and apparatus for selecting a Relay Station (RS) for a Mobile Station (MS) of a communication system are provided. The communication system is provided with the MS, a serving Base Station (BS) for providing the MS with a service and RSs for relaying signals of the MS and the BS. The MS receives RS information for increasing system capacity from the serving BS, measures reference signal strength of an RS indicated in the RS information, and selects an associated RS as a target RS for relaying a signal when the measured reference signal strength satisfies a condition.