A VoIP-based conferencing system readily handles different protocols, load balance media resources and deals with fail-over situations. The VOIP conferencing system has a gateway coupled to the PSTN (Public Switched Telephone Network). A Voice Server Director (VSD) is coupled to the gateway and performs the control function of the conferencing system. The VSD has a back-to-back user agent. The VSD controls a media server, which mixes the audio (signals), and the media server receives the data portion of a conference call from the gateway. The media server is coupled to a bridge that controls an on-going conference.