An encoder for video encoding and decoding with long term reference picture retention. The encoder generates an encoded bitstream for a compliant decoder. At the decoder, coded pictures are decoded using a list of references pictures, of which at least one is a long term reference picture. A first picture is decoded using the long term reference picture. A second picture is decoded without using the long term reference picture which is inactive but is retained in a buffer. A subsequent picture is then decoded again using the long term reference picture.