A video encoder may use an adaptive Wiener filter inside the core video encoding loop to improve coding efficiency. In one embodiment, the Wiener filter may be on the input to a motion estimation unit and, in another embodiment, it may be on the output of a motion compensation unit. The taps for the Wiener filter may be determined based on characteristics of at least a region of pixel intensities within a picture. Thus, the filtering may be adaptive in that it varies based on the type of video being processed.