A method of substitutional neural residual compression is performed by at least one processor and includes estimating motion vectors, based on a current image frame and a previous reconstructed image frame, obtaining a predicted image frame, based on the estimated motion vectors and the previous reconstructed image frame, and subtracting the obtained predicted image frame from the current image frame to obtain a substitutional residual. The method further includes encoding the obtained substitutional residual, using a first neural network, to obtain an encoded representation, and compressing the encoded representation.