A video processing method includes determining, according to a rule, whether to apply a scaling matrix based on whether a secondary transform matrix is applied to a portion of a video block of a video, wherein the scaling matrix is used to scale at least some coefficients of the video block, and wherein the secondary transform matrix is used to transform at least some residual coefficients of the portion of the video block during the conversion; and performing a conversion between the video block of the video and a bitstream representation of the video using the selected scaling matrix.