A method, system and computer program product for classifying an image or video. An image or video to be classified is received. Scene statistics (statistical model of pictures, images or videos representative of pictures, images or videos, respectively, that are captured of the physical world) of the image or video are captured. A model (a statistical model that describes a set of probability distributions) of the image or video is then created using the captured scene statistics. A comparison between the model of the image or video with two other models of images or videos is performed, such as a model of visible light images or videos and a model of infrared images or videos. The received image or video is then classified (e.g., classified as corresponding to a visible light image) based on the comparison.