An image processing device calculates, from a registration image representing a photographed object and three-dimensional shape data in which respective points of a three-dimensional shape of the object are correlated with pixels of the registration image, by assuming uniform albedo, a shadow base vector group having components from which an image under an arbitrary illumination condition can be generated through linear combination. A shadow in the registration image is estimated with using the vector group. A perfect diffuse component image including the shadow is generated, and based on the image a highlight removal image is generated in which a specular reflection component is removed from the registration image. Thus, an image recognition system generates illumination base vectors from the highlight removal image and thereby can obtain the illumination base vectors based on which an accurate image recognition process can be carried out without influence of a specular reflection.