A personal authentication system according to the present invention includes a matrix generation unit, a feature extraction unit, a feature transformation unit, a processing unit, and a data matching unit. The matrix generation unit classifies a plurality of feature amounts of face image data, which are recorded in advance, into classes and calculates as a mapping matrix, coefficients of a linear discriminant equation which uses the recorded plurality of feature amounts as variables. The feature extraction unit extracts a first feature amount from first face image data and extracts a second feature amount from second face image data. The feature transformation unit, by using the mapping matrix, transforms the first feature amount into a first transformed feature amount and transforms the second feature amount into a second transformed feature amount. The processing unit calculates as a similarity, a normalized correlation value between the first transformed feature amount and the second transformed feature amount. The data matching unit judges that the first face image data and the second face image data are image data of same person when the similarity exceeds a predetermined threshold.