A conference control device includes: a communication processing unit which performs transmission/reception processing of conference information containing voice information and image information between the conference control device and a plurality of video conference terminal devices through a network; an information collecting unit which collects information on connection paths between the conference control device and the respective video conference terminal devices; a grouping unit which sorts the plurality of video conference terminal devices into a plurality of groups based on information on the connection paths collected by the information collecting unit; a band monitoring unit which monitors a use band of the communication processing unit in accordance with every group; and a bandwidth control unit which is, when the group having the use band of less than a predetermined value is detected by the band monitoring unit, controls the bandwidth of the video conference terminal devices belonging to the group.