Techniques related to generating downscaled image or image frame data in a luma chroma separated color space for an image or video pipeline architecture are discussed. Such techniques may include converting input image data to the luma chroma separated color space based on adaptive color coefficients determined based on an illumination indicator associated with the input image data and storing downscaled color converted image data to an input image buffer of the image or video pipeline.