Fully scalable encoder and decoder for interlaced video. A method for encoding an interlaced sequence of digital video data decomposes the interlaced video sequence into first and second fields, performs digital filtering to get lower frequency and higher frequency component signals of the first fields, and uses spatio-temporal filtering and motion estimation for generating base layer signals being suitable for reconstruction of a progressive mode video sequence in a receiver. Advantageously, both the spatio-temporal filter at the encoder, and the inverse process at the receiver, can perform scaling in spatial and temporal dimension. The second fields are used to generate enhancement signals, which enable a receiver to reproduce an interlaced video sequence of the full, or scaled, spatial and/or temporal resolution.