A video encoder controlling rate of each frame in a current frame set (Group of Picture, GOP), and method thereof. The video encoder comprises a frame grouping device, a GOP initialization device, and a GOP encoding device. The frame grouping device allocates a target bit budget RGOP to the current frame set. The GOP initialization device is coupled to the frame grouping device, and estimates a first quantization parameter QP1 based on the target bit budget RGOP and complexity of the current frame set. The GOP encoding device is coupled to the GOP initialization device, and encoding a frame in the current frame set with the second quantization parameter QP2 to generate output data.