A method for encoding video signals by inverse motion compensated temporal filtering where video frames of a base layer are used to encode video frames of an enhanced layer into predicted images. For each image block in an arbitrary frame in an enhanced layer frame sequence, an area including a block, which is present in a base layer frame temporally coincident with the arbitrary frame and is at the same position as the image block, is enlarged according to the ratio between screen sizes of the two layers. A reference block most highly correlated with the image block is searched for in the enlarged area in the temporally coincident base layer frame through motion estimation and is used to obtain pixel difference values and a motion vector of the image block for encoding the image block into a predicted image, thereby improving coding efficiency.