A method and a system for patch-based scene segmentation using neural networks are presented. In an embodiment, a method comprises: using one or more computing devices, receiving a digital image comprising test image; using the one or more computing devices, creating, based on the test image, a plurality of grid patches; using the one or more computing devices, receiving a plurality of classifiers that have been trained to identify one or more materials of a plurality of materials; using the one or more computing devices, for each patch of the plurality of grid patches, labelling each pixel of a patch with a label obtained by applying, to the patch, one or more classifiers from the plurality of classifiers; using the one or more computing devices, generating, based on labels assigned to pixels of the plurality of grid patches, a grid of labels for the test image.