The time segment representative feature vector generation device includes an intra-time segment feature vector group selection means for selecting, for each time segment and from a feature vector series for respective frames, feature vectors of a plurality of frames included in a time segment; and a dimension selection means for selecting, for each time segment and from the selected feature vectors of different frames in the time segment, features of different dimensions of the feature vectors, and generating a time segment representative feature vector which is a feature vector representing the time segment.