A method of classifying a multimodal test object described according to at least one first and one second modality is provided, including offline construction by classification of a multimedia dictionary, defined by a plurality of multimedia words, based on a recoding matrix of representatives of the first modality forming a dictionary of the first modality including a plurality of words of the first modality, the recoding matrix constructed to express the frequency of each word of the second modality of a dictionary of the second modality including a plurality of words of the second modality, for each word of the first modality, classification of a multimodal test object performed online by recoding each representative of the first modality relating to the multimedia object considered on the multimedia dictionary base, followed by aggregating representatives of the first modality coded in the recoding in a single vector representative of the multimodal object.