An object recognition system is applicable to practical use, and utilizes image information besides speech information to improve recognition accuracy. The object recognition system comprises a speech recognition unit to determine candidates for a result of speech recognition on input speech and their likelihoods, and an image model generation unit to generate image models of a predetermined number of the candidates having the highest likelihoods. The system further comprises an image likelihood calculation unit to calculate image likelihoods of input images based on the image models, and an object recognition unit to perform object recognition using the image likelihoods. At the time of generating the image model of the candidate, the image model generation unit first searches an image model database, and, when the image model of the candidate is not found in the database, the image model generation unit generates said image model from image information on the web.