In one embodiment, captured video summaries are tagged with metadata so the videos can be easily searched. The videos are classified into different scenes, depending on the type of action in the video, so searching can be based on the type of scene. In one embodiment, tags are provided for moving objects or people. The type of object that is moving is tagged (car, ball, person, pet, etc.). Video search results are ranked based on the weighting of the video events or video summaries. The video summary weighting focuses on important events, with multiple videos/images over a period of time condensed into a short summary video. This creates a weighted video summary with different time-lapse speeds that focuses on important events.