What is disclosed is a system and method for processing image data acquired using a multi-band infrared camera system with a spectral mosaic filter arranged in a geometric pattern without having to perform a demosaicing that is typical with processing data from an array of sensors. In one embodiment, image data that has been captured using a camera system that has a spectral filter mosaic comprising a plurality of spectral filters arrayed on a grid. A material index is determined, using intensity values collected by sensor elements associated with a cell's respective spectral filters. All of the material indices collectively generate a material index image. Thereafter, material identification is performed on the material index image using, for example, pixel classification. Because the demosaicing step can be effectively avoided, image processing time is reduced. The teachings hereof find their uses in a wide array of applications including automated HOV/HOT violation detection.