A neural network is trained for use in a substrate residue classification system by obtaining ground truth residue level measurements of a top layer of a calibration substrate at a plurality of locations, each location at a defined position for a die being fabricated on the substrate. A plurality of color images of the calibration substrate are obtained, each color image corresponding to a region for a die being fabricated on the substrate. A neural network is trained to convert color images of die regions from an in-line substrate imager to residue level measurements for the top layer in the die region.