Interlaced video can be encoded in two layers, base layer and enhancement layer. A method for optimizing the encoding of motion vectors for enhancement layer is proposed. It comprises defining various different ways to encode the motion vectors, estimate the coding costs for each way, and select the way with the lowest coding cost. The various ways to encode enhancement layer motion vectors can be combined. Either motion vectors from base layer are reused, resulting in a scaling factor and an update vector, or new motion vectors are calculated. Either forward or backward or bi-directional prediction can be used. Either one frame or multiple frames can be taken as reference. The reference frame can be from the enhancement layer, base layer or both. Either the complete motion vectors are encoded, or only the difference between the motion vector and the corresponding BL motion vector is encoded.