A system for determining field surface conditions may include a frame member and a ground engaging tool coupled thereto. The ground engaging tool may be configured to engage soil within a field as an agricultural implement is moved across the field. The system may further include a vision sensor having a field of view directed towards a portion of a surface of the field and being configured to capture vision-based data indicative of a field surface condition of the field. The system may also include a secondary sensor coupled to the ground engaging tool and configured to capture secondary data indicative of the field surface condition. Additionally, the system may include a controller configured to determine an initial surface condition as a function of the vision-based data and to adjust the initial surface condition based at least in part on the secondary data received from the secondary sensor.