A conference controller receives access requests to access a conference session from respective callers. Responsive to the requests, the controller sends a conference identifier (ID) and respective agent discovery information to each of the callers. Each caller discovers an appropriate respective media agent based on the respective agent discovery information and sends a join request including the conference ID to that media agent. In turn, the media agents send requests for configuration information to the controller. Responsive to the requests from the media agents, the controller provides configuration information to the media agents that the media agents use to form a media connection with each other for the conference session through which the callers exchange media packets.