A video decoder or encoder can be used to convert and process different video streams having different combinations of frame rate and Chroma fidelity. Rather than setting the maximum sample rate of the encoder based on a Luma sample rate, the encoder's throughput is set based on a maximum color sample rate. Additionally, the picture buffer size can be set based on a maximum number of color pictures. An input of the video decoder receives an input video stream having a given display resolution and encoded in a first format employing a first combination of frame rate and Chroma fidelity. Processing circuitry in the decoder converts the input video stream from the first format to an output video stream having a second format, the second format having the given display resolution and employing a second combination of frame rate and Chroma fidelity different from the first combination.