For saliency mapping, a machine-learned classifier is used to classify input data. A perturbation encoder is trained and/or applied for saliency mapping of the machine-learned classifier. The training and/or application (testing) of the perturbation encoder uses less than all feature maps of the machine-learned classifier, such as selecting different feature maps of different hidden layers in a multiscale approach. The subset used is selected based on gradients from back-projection. The training of the perturbation encoder may be unsupervised, such as using an entropy score, or semi-supervised, such as using the entropy score and a difference of a perturbation mask from a ground truth segmentation.