A block of video data that is split into sub-blocks forms a prediction for that sub-block using predictions from neighboring sub-blocks such that a current prediction for the sub-block is combined with weighted versions of the neighboring predictions. The neighboring sub-blocks motion vectors are checked to determine whether they are different than the motion vector of the sub-block being predicted. If so, than that corresponding neighboring sub-block is used to generate the prediction of the current sub-block. In an embodiment, two lines or two columns of pixels within a sub-block are used in forming the prediction for the current sub-block when the size of the block containing the sub-block is under a particular size.