Embodiments herein disclose a tone mapping system that generates a base layer by filtering a frame in an HDR video. The tone mapping system then generates a tone curve using a tone mapping parameter derived from the base layer - e.g., a mean or maximum luminance value. Once the tone curve parameter is identified, the system performs temporal filtering to smooth out inconsistencies between the current value of the tone curve parameter and the values of tone curve parameter of at least one previous frame in the HDR video. The tone mapping system applies the temporally filtered tone curve to the base layer to generate a temporally coherent layer. The temporally coherent base layer can be combined with a detail layer derived from the HDR video to generate a frame of a tone mapped video.