System and method to summarize one or more videos are provided. The system includes a data receiving module configured to receive videos; a video analysis module configured to analyse the one or more videos to generate one or more transcription text output; a building block data module configured to create a building block model and to apply the building block model on analysed videos; a video presentation module configured to present contents of the videos using elements and to present the one or more transcription texts; a video prioritization configured to generate one or more ranking formulas for the videos, to prioritize building block models, upon receiving feedback from users, based on contents and transcription texts; a video summarization module configured to generate a video summary; a video action module configured to choose an action to be performed on the videos based on the feedback received from the corresponding users.