In one aspect, a system for detecting worn or damaged components of an agricultural machine may include first and second acoustic sensors positioned at first and second locations on the agricultural machine, respectively, with the second location being spaced apart from the first location. A controller of the system may be configured to determine a first acoustic parameter associated with the first location of the agricultural machine based on acoustic data received from the first acoustic sensor. The controller may also be configured to determine a second acoustic parameter associated with the second location of the agricultural machine based on acoustic data received from the second acoustic sensor. Furthermore, the controller may be configured to determine a component of the agricultural machine is worn or damaged when the first acoustic parameter differs from the second acoustic parameter by a predetermined amount.