Systems and methods for generating text corpora comprising realistic optical character recognition (OCR) errors and training language models using the text corpora are provided. An example method comprises: generating, by a computer system, an initial set of images based on an input text corpus comprising text; overlaying, by the computer system, one or more simulated defects over the initial set of images to generate an augmented set of images; generating an output text corpus based on the augmented set of image; and training, using the output text corpus, a language model for optical character recognition.