An orientation estimation apparatus estimates a position and an orientation of an image capturing unit based on positions of a marker on the image and in the real space when the maker is detected; and based on positions of feature points on the captured image and in the real space, and initial values of the position and the orientation of the unit when the maker is not detected. Further, the apparatus calculates a failure index from a first image acquired while the position and the orientation are estimated by the position of marker, where the failure index indicates a possibility that the detection of the marker is failed. When the failure index satisfies a shift-preparation-start reference, the initial values are set from the position and the orientation estimated by the position of the marker on the image having been acquired in a predetermined period before the first image.