A method and apparatus for video encoding or decoding utilizing adaptive background residual prediction is disclosed. The present invention adaptively applies background residual prediction to a current block based on a selection decision. The coding block is split into one or more coding sub-blocks. A reference sub-block in a reference picture is located for a current coding sub-block of the current coding block according to a motion vector associated with the current coding block. A background reference sub-block in a background picture is located for the reference sub-block, where the background reference sub-block is at a first co-located location as the reference sub-block. The method then selects a first predictor or a second predictor to encode or decode the current sub-block based on a selection decision. The first predictor corresponds to the reference sub-block, and the second predictor is derived according to the reference sub-block and the background picture.