A system for detecting material accumulation relative to rotating ground-penetrating tools of an agricultural implement includes an agricultural implement having a frame and at least one rotating ground-penetrating tool supported relative to the frame. The rotating ground-penetrating tool(s) is configured to penetrate into the ground to a given penetration depth. The system also includes a tool speed sensor configured to provide data indicative of a rotational speed of the ground-penetrating tool(s) and a computing system communicatively coupled to the tool speed sensor. The computing system is configured to monitor the rotational speed of the ground-penetrating tool(s) based on the data provided by the tool speed sensor, determine a threshold rotational speed based at least in part on the penetration depth of the ground-penetrating tool(s), and determine when the rotational speed of the ground-penetrating tool(s) falls below the threshold rotational speed.