A method of analyzing a digital video clip to determine candidate frames for subsequent key frame selection, includes: performing a global motion estimate on the video clip that indicates translation of a scene or camera, or scaling of the scene; and forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of camera motion classes. The method further includes extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion. An important feature of the invention is that the rules corresponding to camera motion include extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and not extracting candidate frames for object motion based frames for fast pan and zoom segments, and the rule corresponding to object motion includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length.