Systems and methods are provided for improving efficiency of decoding performance of a video compression system by selectively applying, based on the size of the coding unit, a prediction method, a palette mode, or palette-based residual refinement method which combines the palette mode and the traditional prediction methods comprising deriving a prediction block of a coding unit, decoding palette-based residual information of the coding unit, deriving a palette-based residual block based on the decoded palette-based residual information, and combining the prediction block and the palette-based residual block to construct a reconstructed block.