Systems and methods for implementing a hierarchical image recognition framework for classifying digital images are provided. The provided hierarchical image recognition framework utilizes a multi-layer approach to model training and image classification tasks. A first layer of the hierarchical image recognition framework generates first layer confidence scores, which are utilized by the second layer to produce a final recognition score. The provided hierarchical image recognition framework permits model training and image classification tasks to be performed more accurately and in a less resource intensive fashion than conventional single-layer image recognition frameworks. In some embodiments real-time operator guidance is provided for an image classification task.