A rack server system controlled by a remote controller includes a number of server units and a middle plane board. Each of the server units includes a baseboard management controller (BMC), a server-end connector, and a server-end switch, coupled to the BMC and the server-end connector for regulating communication operation therebetween. The middle plane board includes an integrated connector and a plane-end switch, coupled to the server-end switches of each of the server units and the integrated connector for regulating communication operation thereamong. The remote controller is coupled to the rack server system via one of the integrated connector and the server-end connectors and manages the communication operation of the server units.