Provided is an apparatus and method for determining a transmission mode for a base station in a wireless mobile communication system. The method includes measuring a state of a channel with a mobile station, determining whether channel states between the mobile station and a predetermined number of other mobile stations are orthogonal to each another, determining a space division multiple access (SDMA) mode as the transmission mode for the base station if the channel states are orthogonal to each other, and determining a beamforming (BF) mode as the transmission mode for the base station if the channel states are not orthogonal to each other.