A scalable video compression method and apparatus for encoding an input frame sequence having temporally sequential frames includes partitioning each of the frames of the input frame sequence into a top field and a bottom field to form a base layer sequence of fields and an enhancement layer sequence of fields. The base layer sequence and the enhancement layer sequence each comprise temporally alternating top and bottom fields of the partitioned frames of the input frame sequence. The base layer sequence and the enhancement layer sequence are encoded, wherein at least one field of the enhancement layer sequence is encoded in dependence upon at least one of the fields of the base layer sequence.