A system for managing material accumulation relative to ground engaging tools of an agricultural implement may include a ground engaging tool and an acoustic sensor configured to generate data indicative of an acoustic parameter of a sound produced as the ground engaging tool engages the ground when a ground engaging operation is performed within a field. Additionally, the system may include a controller communicatively coupled to the acoustic sensor. The controller may be configured to monitor the data received from the acoustic sensor and determine a presence of material accumulation relative to the ground engaging tool based at least in part on the acoustic parameter.