Invention Grant
- Patent Title: Method of and system for explainable knowledge-based visual question answering
-
Application No.: US17130599Application Date: 2020-12-22
-
Publication No.: US11599749B1Publication Date: 2023-03-07
- Inventor: Maryam Ziaeefard , Freddy Lecue
- Applicant: THALES SA
- Applicant Address: FR Courbevoie
- Assignee: THALES SA
- Current Assignee: THALES SA
- Current Assignee Address: FR Courbevoie
- Agency: Hauptman Ham, LLP
- Main IPC: G06K9/62
- IPC: G06K9/62 ; G06N3/04 ; G06N3/08 ; G06V10/40 ; G06V30/262

Abstract:
A method and a system for generating an augmented scene graph of an image and for training an explainable knowledge based (KB) visual question answering (VQA) machine learning (ML) model are provided. A scene graph encoding spatial and semantic features of objects and relations between objects in the image is obtained. An augmented scene graph is generated by embedding a knowledge graph to enhance the scene graph. An embedded set of questions and associated answers related to the image are obtained. The KB VQA ML model is trained to provide an answer to a given question related to the image based on the augmented scene graph and the embedded set of questions and associated answers. The KB VQA ML model is trained to retrieve a subgraph linking the question and the associated answer as a potential explanation for the answer.
Information query