Blurriness in a video image transmitted over a lossy network can be reduced by filtering the decoder output to determine whether updating of the reference frame based on the current video data is indicated. An update may be appropriate if the decoder output is sharper than the contents of the display buffer after the last update. Additionally, an update may be appropriate if there was a significant change to the underlying image, and thus to the encoder's input. Whether such a change has occurred can be determined by analyzing the prediction mode, motion vectors, and pixel data of the output image.