An operation detection method of detecting an operation of a pointing element with respect to an operation surface includes converting first and second taken image obtained by imaging the operation surface into first and second converted taken image calibrated with respect to the operation surface respectively, determining whether or not the pointing element contacts with the operation surface and is in a resting state based on the first and second converted taken image, selecting the first and second converted taken image at when it was determined that the pointing element contacts and to be in the resting state as first and second reference image, calculating a first difference in position between the pointing element in the first reference image and the pointing element in the first converted taken image, calculating a second difference in position between the pointing element in the second reference image selected and the pointing element in the second converted taken image, and determining whether or not the pointing element separated from the operation surface based on the first difference and the second difference.