Techniques related to designating golden frames and to determining frame sizes and/or quantization parameters golden and non-golden frames for video coding are discussed. Such techniques may include designating an frame as a golden frame or a non-golden frame based on whether the frame is a scene change frame, a distance of the frame to a previous golden frame, and an average temporal distortion of the frame and determining a frame size and/or quantization parameter for the frame based on the designation and a temporal distortion of the frame.