In one aspect, the present subject matter is directed to a system for managing material accumulation relative to a ground engaging assembly of an agricultural implement. The system may include one or more ground engaging tools of the ground engaging assembly, a sensor detecting material accumulation relative to the ground engaging tools, and a controller communicatively coupled to the sensor. The controller may receive an input from the sensor associated with material accumulation relative to the ground engaging tools. The controller may further access data associated with an amount of crop residue present and data associated with a moisture content of soil within the field. Additionally, the controller may determine an accumulation type of the material accumulation based at least in part on the crop residue data and the soil moisture data.