The disclosure includes a system and method for decomposing a video to salient fragments and synthesizing a video composition based on the salient fragments. A video decomposition application extracts non-salient portions of a video, extracts a plurality of salient fragments of the video, builds a database of the plurality of salient fragments, receives a query, retrieves, from the database of the plurality of salient fragments, a set of salient fragments based on the query, and synthesizes a video composition based on the set of salient fragments and the non-salient portions of the video.