A method of adjusting brightness of a foreground video displayed over a background in a user device is provided. The method includes determining enhancement factors for background sections in a background frame of the background, determining an enhancement mask for each foreground sub-frame of the foreground video based on enhancement factor of corresponding background section and determining enhanced brightness values for each foreground pixel in the foreground video based on the enhancement mask. The method of the disclosure provides real-time, clear and distinct viewing of an enhanced foreground video in varying brightness conditions of the background sections.