A linear transformation matrix calculating apparatus linearly transforms a plurality of dictionary subspaces which belong to respective categories by a linear transformation matrix respectively, selects a plurality of sets of two dictionary subspaces from the plurality of linearly transformed dictionary subspaces, calculates a loss function using similarities among the selected sets of dictionary subspaces respectively, calculates a differential parameter obtained by differentiating the loss function by the linear transformation matrix, calculates a new linear transformation matrix from the differential parameter and the linear transformation matrix by Deepest Descent Method, and updates the new linear transformation matrix as the linear transformation matrix used in the linear transformation unit.