In one aspect, a system for controlling an operation of a tillage implement being towed across a field by a work vehicle may include a tillage tool configured to engage soil and crop material present within a field as the tillage implement is being towed across the field by the work vehicle. Furthermore, the system may include a controller configured to receive an input associated with crop material present within the field and determine a location of a crop row relative to the tillage tool based on the received input. Additionally, the controller may be configured to control a direction of travel of the tillage implement based on the determined location of the crop row.