Described herein are systems, methods, and other techniques for controlling a velocity of a construction machine operating within a construction site. Sensor data is captured using one or more sensors of the construction machine while the construction machine is moving at the velocity in a forward or a backward direction. An actual surface of the construction site is estimated based on the sensor data. A deviation between a target surface and the actual surface is calculated. An actual performance metric is calculated based on the deviation. The actual performance metric is compared to a target performance metric to determine a velocity adjustment. The velocity of the construction machine is adjusted by the velocity adjustment.