The present disclosure relates to systems, methods, and non-transitory computer readable media for efficiently, quickly, and flexibly generating and providing pixel-wise classification predictions utilizing early exit heads of a multi-exit pixel-level prediction neural network. For example, the disclosed systems utilize a multi-exit pixel-level prediction neural network to generate classification predictions for a digital image on the pixel level. The multi-exit pixel-level prediction neural network includes a specialized architecture with early exit heads having unique encoder-decoder architectures for generating pixel-wise classification predictions at different early exit stages. In some embodiments, the disclosed systems implement a spatial confidence-adaptive scheme to mask certain predicted pixels to prevent further processing of the masked pixels and thereby reduce computation.