Invention Grant
- Patent Title: System and method for synthesizing photo-realistic video of a speech
-
Application No.: US17019203Application Date: 2020-09-12
-
Publication No.: US11682153B2Publication Date: 2023-06-20
- Inventor: Chao Pan , Wenbo Liu , Lei Yi
- Applicant: JINGDONG DIGITS TECHNOLOGY HOLDING CO., LTD. , JD FINANCE AMERICA CORPORATION
- Applicant Address: CN DE Beijing
- Assignee: JINGDONG DIGITS TECHNOLOGY HOLDING CO., LTD.,JD FINANCE AMERICA CORPORATION
- Current Assignee: JINGDONG DIGITS TECHNOLOGY HOLDING CO., LTD.,JD FINANCE AMERICA CORPORATION
- Current Assignee Address: CN Beijing; US DE Wilmington
- Agency: Scully, Scott, Murphy & Presser, P.C.
- Main IPC: G10L13/00
- IPC: G10L13/00 ; G06T13/20 ; G06N3/049 ; G10L13/04 ; G06T13/40 ; G06V40/10

Abstract:
A system and a method for obtaining a photo-realistic video from a text. The method includes: providing the text and an image of a talking person; synthesizing a speech audio from the text; extracting an acoustic feature from the speech audio by an acoustic feature extractor; and generating the photo-realistic video from the acoustic feature and the image by a video generation neural network. The video generating neural network is pre-trained by: providing a training video and a training image; extracting a training acoustic feature from training audio of the training video by the acoustic feature extractor; generating video frames from the training image and the training acoustic feature by the video generation neural network; and comparing the generated video frames with ground truth video frames using generative adversarial network (GAN). The ground truth video frames correspond to the training video frames.
Public/Granted literature
- US20220084273A1 SYSTEM AND METHOD FOR SYNTHESIZING PHOTO-REALISTIC VIDEO OF A SPEECH Public/Granted day:2022-03-17
Information query