A paper sheet processing apparatus includes: an insertion slot through which a paper sheet is inserted; a sensor which detects insertion of the paper sheet; a traveling route through which the paper sheet is conveyed; a light emitting part which irradiates the traveling route with light; a light receiving part which receives light from the traveling route; a black calibration part which sets a reference value for the lowest brightness in a state that the emission of the light emitting part is ineffective; and an authenticity judgment processing part which judges an authenticity. The black calibration part can set the reference value before the authenticity judgment processing part judges the authenticity after the sensor detects the insertion of the paper sheet at every time when the paper sheet is inserted.