A method for temporal decomposition and reconstruction of an input video signal is disclosed. The method uses a prediction process and an update process in the framework of motion compensated temporal filtering (MCTF), the motion information used for the update process being derived from the motion information used for the prediction process, and the method employing a block based video codec composed of an encoder and a decoder. In response to the update and prediction processes on a picture being based on N reference pictures where N is greater than one, the update process is split into N update processes using a single picture as a reference. Each one of the split update processes is executed each time a picture that is needed as a reference by the process is received.