A method determines similarity of objects depicted in images when the images pertain to different modalities. The method includes obtaining images that depict the objects and that pertain to the different modalities. An embedding function is applied to each of the images. The embedding function is selected from a set of two or more embedding functions, each of the embedding functions corresponding to a modality of the different modalities, the selected embedding function corresponding to the modality of the image to which it is applied. Application of the embedding function maps that image to a representation in a representation space such that when the images are mapped to the representation space, a distance between the representations of the images is indicative of a similarity of their depicted objects. The similarity of the depicted objects is determined based on the location of the corresponding representations in the representation space.