In one aspect, a method for detecting residue bunches within a field includes receiving one or more images depicting an imaged portion of an agricultural field, with the imaged portion of the field being at least partially represented by a plurality of pixels within the one or more images. The method also includes classifying a portion of the plurality of pixels that are associated with crop residue within the imaged portion of the field as residue pixels, with each residue pixel being associated with a respective pixel height. Additionally, the method includes identifying each residue pixel having a pixel height that exceeds a pixel height threshold as a candidate residue bunch pixel, and determining whether a residue bunch is present within the imaged portion of the field based at least in part on the candidate residue bunch pixels.