The present invention relates to an auto-focus system and method based on a sensor, using hierarchical extraction and motion estimation for an image. The auto-focus method may include the steps of: (a) extracting a phase difference image from a sensed image; (b) extracting a robust feature image which satisfies a preset reference with respect to noise and out-of-focus blur, from the phase difference image; and (C) estimating a motion vector for lens shift from the feature image.