The shape measurement apparatus calculates a characteristic amount for a plurality of points of interest on a surface of a measurement target object, based on an image obtained by image capturing with a camera, calculates an orientation of a normal line based on a value of the characteristic amount by referencing data stored in advance in a storage device, and restores the three-dimensional shape of the surface of the measurement target object based on a result of the calculation. The storage device stores a plurality of data sets generated respectively for a plurality of reference positions arranged in a field of view of the camera, and the data set to be referenced is switched depending on a position of a point of interest.