A first improvement is described for prediction of motion vectors to be used in prediction of video data for enhancement layer data. Arbitrary pixelblock partitioning between base layer data and enhancement layer data raises problems to identify base layer motion vectors to be used as prediction sources for enhancement layer motion vectors. The disclosed method develops enhancement layer motion vectors by scaling a base layer pixelblock partition map according to a size difference between the base layer video image and the enhancement layer video image, then identified scale base layer pixelblocks that are co-located with the enhancement layer pixelblocks for which motion vector prediction is to be performed. Motion vectors from the scaled co-located base layer pixelblocks are averaged, weighted according to a degree of overlap between the base layer pixelblocks and the enhancement layer pixelblock. Another improvement is obtained by filtering recovered base layer image data before being provided to an enhancement layer decoder. When a specified filter requires image data outside a prediction region available from a base layer decoder, the prediction region data may be supplemented with previously-decoded data from an enhancement layer at a border of the prediction region.