A method and apparatus for palette coding of a block of video data using a shared palette/palette prediction buffer are disclosed. The system receives a bitstream including compressed data associated with a current palette coded block and determines a palette predictor for the current palette coded block stored in a shared palette/palette predictor buffer. Reuse flags for entries of the current palette are used to update the current palette stored in the shared palette/palette predictor buffer. Any new signaled entry will be inserted to the shared palette/palette predictor buffer. An updated palette predictor for a next palette coded block is generated accordingly. A method of palette management and initialization for first palette-coded block in a high-level video structure is also disclosed. The high-level video structure may correspond to a slice, wavefront, sequence or picture.