Devices, systems and methods for digital video coding, which includes matrix-based intra prediction methods for video coding, are described. In a representative aspect, a method for video processing includes performing a first determination that a luma video block of a video is coded using a matrix based intra prediction (MIP) mode in which a prediction block of the luma video block is determined by performing, on previously coded samples of the video, a boundary downsampling operation, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation, performing, based on the first determination, a second determination about a chroma intra mode to be used for a chroma video block associated with the luma video block, and performing, based on the second determination, a conversion between the chroma video block and a bitstream representation of the chroma video block.