Imaging systems with image sensors and image processing circuitry are provided. The image processing circuitry may identify motion and perform autofocus (e.g., continuous autofocus) using images captured by an image sensor. Auto exposure metrics such as average luminance values and autofocus statistics such as sharpness scores may be calculated for each image. The auto exposure metrics may be used to calculate motion scores and identify directional motion between a series of captured images. The motion scores may be used with the sharpness scores to determine when to perform autofocus functions such as when to refocus a lens for a continuous autofocus application. For example, the motion scores may be monitored to identify motion that exceeds a given magnitude and duration. After identification of motion, motion scores and sharpness scores may be used to determine when a given scene has stabilized and when the lens should be refocused.