An example device for decoding video data includes a memory for storing the video data and one or more processors implemented in circuitry and communicatively coupled to the memory. The one or more processors are configured to determine whether a first coding unit (CU) is a skip mode CU, and based on the first CU not being a skip mode CU, determine whether the first CU is encoded using one of an intra mode or a palette mode. The one or more processors are also configured to determine whether the first CU is encoded using the palette mode based on the first CU being encoded using one of the intra mode or the palette mode. The one or more processors are also configured to decode the first CU based on the determination of whether the first CU is encoded using the palette mode.