To detect a fluorescent light and a phosphorescent light emitted from a paper sheet with a high precision, a fluorescence and phosphorescence detecting apparatus includes two sensor units. Each sensor unit includes a light source that emits an ultraviolet light on the paper sheet, an image sensor that captures images of the fluorescent light and the phosphorescent light excited on the paper sheet by irradiation of the ultraviolet light, and a light receiving lens that guides the fluorescent light and the phosphorescent light excited on the paper sheet to the image sensor. The two sensor units are arranged above and below the transport path and opposing each other across the transport path, and the light receiving lenses and the image sensors in the two sensor units are off-set along a transport direction of the paper sheet.