Multiple pixels are selected from one or more images of an object fabricated from an unknown material captured from one or more viewing directions and one or more trained classification engines are applied to the selected pixels so as to obtain initial estimates of the material at each selected pixel. The one or more trained classification engines are each trained at a training direction and are each characterized by performance characteristics that vary based at least in part on directional deviation from that training direction. A posterior probability for the collection of selected multiple pixels is calculated, wherein the posterior probability is calculated based at least in part on the performance characteristics and directional deviation of a surface normal of the object at a selected pixel from the one or more training directions. The material of the object is classified based on the calculated posterior probability.