In one aspect, a system for controlling the operation of a residue removal device of a seed-planting implement may include a residue removal device configured to remove residue from a path of the seed-planting implement. The system may also include a sensor configured to capture data indicative of a residue characteristic associated with a portion of the field within a detection zone positioned forward of the residue removal device relative to a direction of travel of the seed-planting implement. Furthermore, the system may include a controller communicatively coupled to the sensor. As such, the controller may be configured to monitor the residue characteristic associated with the portion of the field within the detection zone based on data received from the sensor. Additionally, the controller may be further configured to control the operation of the residue removal device based on the monitored residue characteristic.