Method and apparatus for using block size restrictions in video decoding are described. In a representative aspect, a method for video decoding includes receiving a bitstream representation of video data including a block wherein the block comprises samples associated with a first component and second components, wherein samples associated with the first component of the block have a dimension M×N; and decoding the first component and the second components of the block; wherein decoding the first component of the block comprises, based on the dimension, decoding a plurality of sub-blocks for the first component of the block, and the plurality of the sub-blocks are generated by performing a splitting operation only on the samples associated with the first component of the block and not on the samples associated with the second components of the block.