A computerized method and system of segment-based video encoding optimization, the method comprising obtaining a set of input segments constituting an input video stream; selecting one or more input segments in accordance with a selection criterion to be one or more representative segments; determining an encoding instruction and obtaining one or more encoded representative segments encoded from the representative segments in accordance with the encoding instruction and using a quality-driven video encoding scheme; determining, based on the encoded representative segments, an encoding configuration, and obtaining one or more encoded input segments encoded from at least a sub-set of the input segments using the encoding configuration; determining an evaluation instruction used to evaluate quality for each encoded input segment; and generating an output video stream, by selecting, for each input segment, a corresponding output segment from a group comprising at least one of the following candidate segments: the input segment, an encoded representative segment, and an encoded input segment.