A computer-implemented method and system for reconstructing a clean document from annotated document images and/or extracting annotations therefrom are provided. The method includes receiving a set of at least two annotated document images into computer memory, selecting a representative image from the set of annotated document images, performing a global alignment on each of the set of annotated document images with respect to the selected representative image, and forming a consensus document image based at least on the aligned annotated document images. A clean document based at least on the consensus document image is then formed which can be used for extracting the annotations.