Disclosed is a method for video encoding and decoding using adaptive color transform on a residual signal. In particular, the maximum memory size required for adaptive color transform is limited by: controlling whether to allow color space conversion on a sequence level according to a syntax element indicating the maximum conversion size allowed for a sequence of pictures; or controlling whether to apply color space conversion on a block level according to the size of a coding block.