Invention Grant
- Patent Title: Spatio-temporal graph for video captioning with knowledge distillation
-
Application No.: US16827252Application Date: 2020-03-23
-
Publication No.: US11604936B2Publication Date: 2023-03-14
- Inventor: Boxiao Pan , Haoye Cai , De-An Huang , Kuan-Hui Lee , Adrien David Gaidon , Ehsan Adeli-Mosabbeb , Juan Carlos Niebles Duque
- Applicant: TOYOTA RESEARCH INSTITUTE, INC. , THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
- Applicant Address: US CA Stanford
- Assignee: TOYOTA RESEARCH INSTITUTE, INC.,THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
- Current Assignee: TOYOTA RESEARCH INSTITUTE, INC.,THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
- Current Assignee Address: US CA Stanford
- Agency: Seyfarth Shaw LLP
- Main IPC: G06K9/62
- IPC: G06K9/62 ; G06V20/56 ; G06V10/00

Abstract:
A method for scene perception using video captioning based on a spatio-temporal graph model is described. The method includes decomposing the spatio-temporal graph model of a scene in input video into a spatial graph and a temporal graph. The method also includes modeling a two branch framework having an object branch and a scene branch according to the spatial graph and the temporal graph to learn object interactions between the object branch and the scene branch. The method further includes transferring the learned object interactions from the object branch to the scene branch as privileged information. The method also includes captioning the scene by aligning language logits from the object branch and the scene branch according to the learned object interactions.
Public/Granted literature
- US20210295093A1 SPATIO-TEMPORAL GRAPH FOR VIDEO CAPTIONING WITH KNOWLEDGE DISTILLATION Public/Granted day:2021-09-23
Information query