A hybrid-based image clustering method and a server for operating the same are provided. The method includes extracting a video object from each of a plurality of videos, calculating information about a similar key point between a predetermined reference target object and the extracted video object, clustering at least some of the plurality of videos into a first video group associated with the reference target object based on the information about the similar key point, and clustering at least some of the videos included in the first video group into a second video group based on at least one of video capture location information and place tag information of the videos included in the first video group.