公开(公告)号：US11238631B2

公开(公告)日：2022-02-01

申请号：US16855362

申请日：2020-04-22

Applicant: SRI International

Inventor： Karan Sikka ,  Ajay Divakaran ,  Samyak Datta

IPC: G06T11/60 ,  G06F16/56 ,  G06F16/51

Abstract: A method, apparatus and system for visual grounding of a caption in an image include projecting at least two parsed phrases of the caption into a trained semantic embedding space, projecting extracted region proposals of the image into the trained semantic embedding space, aligning the extracted region proposals and the at least two parsed phrases, aggregating the aligned region proposals and the at least two parsed phrases to determine a caption-conditioned image representation and projecting the caption-conditioned image representation and the caption into a semantic embedding space to align the caption-conditioned image representation and the caption. The method, apparatus and system can further include a parser for parsing the caption into the at least two parsed phrases and a region proposal module for extracting the region proposals from the image.

公开(公告)号：US20210056742A1

公开(公告)日：2021-02-25

申请号：US16855362

申请日：2020-04-22

Applicant: SRI International

Inventor： Karan Sikka ,  Ajay Divakaran ,  Samyak Datta

IPC: G06T11/60 ,  G06F16/51 ,  G06F16/56

Abstract: A method, apparatus and system for visual grounding of a caption in an image include projecting at least two parsed phrases of the caption into a trained semantic embedding space, projecting extracted region proposals of the image into the trained semantic embedding space, aligning the extracted region proposals and the at least two parsed phrases, aggregating the aligned region proposals and the at least two parsed phrases to determine a caption-conditioned image representation and projecting the caption-conditioned image representation and the caption into a semantic embedding space to align the caption-conditioned image representation and the caption. The method, apparatus and system can further include a parser for parsing the caption into the at least two parsed phrases and a region proposal module for extracting the region proposals from the image.