In some embodiments, a method receives an encoded video and decodes blocks in a frame of the encoded video in which replacement content will be inserted. The blocks are analyzed to determine first blocks of the frame that are affected by insertion of the replacement content and second blocks in the frame that are not affected by the insertion. Information from the decoding of the block is saved for the second blocks. The frame is encoded where the replacement content is inserted to replace the first blocks in the encoding of the frame, and wherein the information from the decoding of the second blocks is reused when performing the encoding of the second blocks.