Systems and methods are disclosed for processing an image to detect anomalous pixels. An image classification is received from a trained convolutional neural network (CNN) for an input image with a positive classification being defined to represent detection of an anomaly in the image and a negative classification being defined to represent absence of an anomaly. A backward propagation analysis of the input image for each layer of the CNN generates an attention mapping that includes a positive attention map and a negative attention map. A positive mask is generated based on intensity thresholds of the positive attention map and a negative mask is generated based on intensity thresholds of the negative attention map. An image of segmented anomalous pixels is generated based on an aggregation of the positive mask and the negative mask.