A method is provided for providing automated testing of an Optical Character Recognition (OCR) system. An automated testing framework may convert original text files to images of various formats and resolutions. The images may comprise various fonts and layouts according to a document type. The images may be processed by the OCR system to generate a converted text file. Converted text files may be compared to original text files and an OCR accuracy score may be calculated.