In one aspect, a system for determining soil clod size distribution as an agricultural implement is being towed across a field by a work vehicle may include a vision-based sensor provided in operative association with one of the work vehicle or the agricultural implement. As such, the vision-based sensor may be configured to capture vision data associated with a portion of the field present within a field of view of the vision-based sensor. Furthermore, the system may include a controller configured to receive vision data from the vision-based sensor. Moreover, the controller may be further configured to analyze the received vision data using a spectral analysis technique to determine a size distribution of soil clods present within the field of view of the vision-based sensor.