A video transmission system and the method of operation thereof includes: a video transmission unit for receiving a first video frame from an input device, the first video frame having base frame parameters; dividing a color gamut into uniform regions for collecting color data from pixels of the base frame parameters; collecting pixel statistics from each of the uniform regions from the base frame parameters; determining chroma partition coordinates from the pixel statistics; deriving a search pattern of search points based on the chroma partition coordinates; and selecting the search point from the search pattern for color mapping of the first video frame.