A paper sheet recognition apparatus recognizes a type of a paper sheet in an input image by matching the input image of the paper sheet with reference images of a plurality of paper sheets. The apparatus includes a candidate selecting unit that selects a predetermined number of candidate types of the paper sheet based on a density feature and direction features of each block acquired by uniformly dividing the input image and the reference images; and a detailed judgment unit that adaptively divides the input image into blocks according to features of a reference image corresponding to each type of paper sheets selected by the candidate selecting unit, and performs a detailed judgment based on matching values between corresponding blocks of the divided input image and each of the reference images.