A computer-implementable method for creating a summary video with depth information, comprising: recognizing moving objects from the input original video; generating an animated moving object cutout for each of the recognized moving objects by duplicating and stacking successive frames in the input original video containing images of the respective moving object; constructing a scene background by using texture of scenes in the input original video and estimating any missing portion; rendering a Dynamic 3D Scene by using depth information of foreground objects in the input original video and the scene background to create a three-dimensional scene and overlaying the animated moving object cutouts on to the three-dimensional scene according to their respective longitude, latitude, and depth locations in the three-dimensional scene; and synthesizing the summary video, by using the Dynamic 3D Scene.