An image signal processing apparatus includes a detecting unit configured to detect a motion vector of a tracking point provided in an object in a moving image, a computing unit configured to compute a reliability parameter representing the reliability of the detected motion vector, a determining unit configured to determine whether the detected motion vector is adopted by comparing the computed reliability parameter with a boundary, an accumulating unit configured to accumulate the reliability parameter, and a changing unit configured to change the boundary on the basis of the accumulated reliability parameters.