A system and method for extracting structured information from image documents is disclosed. An input image document is obtained, and the input image document may be analyzed to determine a skeletal layout of information included in the input image document. A measure of similarity between the determined skeletal layout and each of the document templates may be determined. A document template may be selected as a matched template, based on the determined measure of similarity. Box areas from the input image document may be cropped out, and optical character recognition (OCR) may be performed on the box areas. Obtained recognized text may be automatically processed using directed search to correct errors made by the OCR. Statistical language modeling may be used to classify the input image document into a classification category, and the classified input image document may be processed according to the classification category.