A position and orientation measurement device includes a grayscale image input unit that inputs a grayscale image of an object, a distance image input unit that inputs a distance image of the object, an approximate position and orientation input unit that inputs an approximate position and orientation of the object with respect to the position and orientation measurement device, and a position and orientation calculator that updates the approximate position and orientation. The position and orientation calculator calculates a first position and orientation so that an object image on an image plane and a projection image of the three-dimensional shape model overlap each other, associates the three-dimensional shape model with the image features of the grayscale image and the distance image, and calculates a second position and orientation on the basis of a result of the association.