A system and method for encoding a video is disclosed having a plurality of frames with spatially correspondent macroblocks. The method includes selecting a coding mode for each spatially correspondent macroblock, such as intra-frame coding or inter-frame coding. A weighting value is determined for each pixel in the macroblock based on the coding mode of the macroblock or other indication of the correlation of the pixel to spatially corresponding pixels in one or more other frames. A temporal filter uses the weighting values to filter noise from the plurality of frames and the resulting noise-reduced output is used in a predictive reference frame.