Systems, devices, and methods for material recognition extract one or more features from a patch in an image of a scene; generate an initial prediction of a material category of the patch from candidate material categories based on the one or more features; identify a scene in the image; identify one or more objects in the image; obtain a first relationship model of two or more of the one or more features, the candidate material categories, the one or more objects, and the scene; and make a refined prediction of the material category of the patch based on the initial prediction of the material category and on the first relationship model.