A system is provided for controlling operation of an engine brake system of an engine in a vehicle. The system includes a controller having an over-speed condition detection unit and an operation mode transition unit. The over-speed condition detection unit is configured to detect an over-speed condition based on a current engine speed and a fuel cut limit speed, the fuel cut limit speed being a predetermined engine speed at which fuel supplied to the engine is suspended. The operation mode transition unit is configured to control the operation of the engine brake system by transitioning the controller between a plurality of brake operation modes based on at least one transition parameter.