A video coder that implements illumination compensation is provided. The video coder receives a first block of pixels in a first video picture to be coded as a current block, wherein the current block is associated with a motion vector that references a second block of pixels in a second video picture as a reference block. The video coder performs inter-prediction for the current block by using the motion vector to generate a set of motion-compensated pixels for the current block. The video coder modifies the set of motion-compensated pixels of the current block by applying a linear model that is computed based on neighboring samples of the reference block and of the current block. The neighboring samples are identified based on a position of the current block within a larger block.