The embodiments herein describe a forecasting system that uses captured images or a location to generate a weather forecast for that location. As used herein, a hyper-location is any location where images of that location are available to the forecasting system. For example, a hyper-location can be an airport where a security camera can provide historical images of the weather conditions at the airport. In one embodiment, the forecasting system can extract attributes from the images that indicate the historical weather conditions at the hyper-location. The forecasting system can then use those weather conditions to select which one of a plurality of historical scenarios best matches the weather conditions. The selected scenario can then be used to train a machine learning (ML) model that tunes a weather forecast for that location.