公开(公告)号：US20220165171A1

公开(公告)日：2022-05-26

申请号：US17535675

申请日：2021-11-25

Applicant: University of Electronic Science and Technology of China

Inventor： Xing XU ,  Jingran ZHANG ,  Fumin SHEN ,  Jie SHAO ,  Hengtao SHEN

IPC: G09B5/06 ,  G06V10/44 ,  G06V10/26

Abstract: The disclosure provides a method for enhancing audio-visual association by adopting self-supervised curriculum learning. With the help of contrastive learning, the method can train the visual and audio model without human annotation and extracts meaningful visual and audio representations for a variety of downstream tasks in the context of a teacher-student network paradigm. Specifically, a two-stage self-supervised curriculum learning scheme is proposed to contrast the visual and audio pairs and overcome the difficulty of transferring between visual and audio information in the teacher-student framework. Moreover, the knowledge shared between audio and visual modality serves as a supervisory signal for contrastive learning. In summary, with the large-scale unlabeled data, the method can obtain a visual and an audio convolution encoder. The encoders are helpful for downstream tasks and cover the training shortage causing by limited data.

公开(公告)号：US20220138495A1

公开(公告)日：2022-05-05

申请号：US17519604

申请日：2021-11-05

Applicant: University of Electronic Science and Technology of China

Inventor： Xing XU ,  Yangye FU ,  Yang YANG ,  Jie SHAO ,  Zheng WANG

IPC: G06K9/62 ,  G06N3/04 ,  G06N3/08

Abstract: A multi-source domain adaptation model by aligning partial features includes a general feature extraction module, a feature selection module for partial feature extraction with the dedicated loss function, three partial feature alignment losses, and two classifiers for adversarial training, where the three partial feature alignment losses include an intra-class partial feature alignment loss, an inter-domain partial feature alignment loss, and an inter-class partial feature alignment loss. With the partial features extracted by the general feature extraction module and the feature selection module following three different partial feature alignment losses, the model is capable of clustering the samples from the identical categories and isolating samples from distinct classes.