A speech recognizer control system, a speech recognizer control method, and a speech recognizer control program make it possible to properly identify a device on the basis of a speech utterance of a user and to control the identified device. The speech recognizer control system includes a speech input unit to which a speech utterance is input from a user, a speech recognizer which recognizes the content of the input speech utterance, a device controller which identifies a device to be controlled among a plurality of devices on the basis of at least the recognized speech utterance content and which controls an operation of the identified device, and a state change storage which stores, as first auxiliary information for identifying a device to be controlled, a state change other than at least a state change caused by a speech utterance from the user among the state changes of operations in the individual devices of the plurality of devices. The device controller carries out first device identification processing for identifying the device to be controlled on the basis of the recognized speech utterance content, and if the device cannot be identified by the first device identification processing, then the device controller uses at least the first auxiliary information to identify the device.