公开(公告)号：US11468544B2

公开(公告)日：2022-10-11

申请号：US17355687

申请日：2021-06-23

Applicant: Snap Inc.

Inventor： Chen Cao ,  Wen Zhang ,  Menglei Chai ,  Linjie Luo

IPC: G06T5/00 ,  G06T5/50 ,  G06T7/40 ,  G06V40/18

Abstract: Systems, devices, media, and methods are presented for generating texture models for objects within a video stream. The systems and methods access a set of images as the set of images are being captured at a computing device. The systems and methods determine, within a portion of the set of images, an area of interest containing an eye and extract an iris area from the area of interest. The systems and methods segment a sclera area within the area of interest and generate a texture for the eye based on the iris area and the sclera area.

公开(公告)号：US20220036647A1

公开(公告)日：2022-02-03

申请号：US17499659

申请日：2021-10-12

Applicant: Snap Inc.

Inventor： Soumyadip Sengupta ,  Linjie Luo ,  Chen Cao ,  Menglei Chai

IPC: G06T17/00 ,  G06T15/04 ,  G06T15/80

Abstract: A shape generation system can generate a three-dimensional (3D) model of an object from a two-dimensional (2D) image of the object by projecting vectors onto light cones created from the 2D image. The projected vectors can be used to more accurately create the 3D model of the object based on image element (e.g., pixel) values of the image.

公开(公告)号：US20210407163A1

公开(公告)日：2021-12-30

申请号：US17364218

申请日：2021-06-30

Applicant: Snap Inc.

Inventor： Menglei Chai ,  Jian Ren ,  Aliaksandr Siarohin ,  Sergey Tulyakov ,  Oliver Woodford

IPC: G06T13/00 ,  G06T7/20 ,  G06T11/00 ,  G06N3/04 ,  G06N3/08

Abstract: Systems and methods herein describe novel motion representations for animating articulated objects consisting of distinct parts. The described systems and method access source image data, identify driving image data to modify image feature data in the source image sequence data, generate, using an image transformation neural network, modified source image data comprising a plurality of modified source images depicting modified versions of the image feature data, the image transformation neural network being trained to identify, for each image in the source image data, a driving image from the driving image data, the identified driving image being implemented by the image transformation neural network to modify a corresponding source image in the source image data using motion estimation differences between the identified driving image and the corresponding source image, and stores the modified source image data.

公开(公告)号：US20210319540A1

公开(公告)日：2021-10-14

申请号：US17355687

申请日：2021-06-23

Applicant: Snap Inc.

Inventor： Chen Cao ,  Wen Zhang ,  Menglei Chai ,  Linjie Luo

IPC: G06T5/00 ,  G06K9/00 ,  G06T5/50 ,  G06T7/40

Abstract: Systems, devices, media, and methods are presented for generating texture models for objects within a video stream. The systems and methods access a set of images as the set of images are being captured at a computing device. The systems and methods determine, within a portion of the set of images, an area of interest containing an eye and extract an iris area from the area of interest. The systems and methods segment a sclera area within the area of interest and generate a texture for the eye based on the iris area and the sclera area.

公开(公告)号：US12277639B2

公开(公告)日：2025-04-15

申请号：US18149007

申请日：2022-12-30

Applicant: Snap Inc.

Inventor： Aleksandr Belskikh ,  Menglei Chai ,  Antoine Chassang ,  Anna Kovalenko ,  Pavel Savchenkov

IPC: G06T13/40 ,  G06N3/094 ,  G06T19/00

Abstract: Embodiments enable virtual hair generation. The virtual hair generation can be performed by generating a first image of a face using a GAN model, applying 3D virtual hair on the first image to generate a second image with 3D virtual hair, projecting the second image with 3D virtual hair into a GAN latent space to generate a third image with virtual hair, performing a blend of the virtual hair with the first image of the face to generate a new image with new virtual hair that corresponds to the 3D virtual hair, training a neural network that receives the second image with the 3D virtual hair and provides an output image with virtual hair, and generating using the trained neural network, a particular output image with hair based on a particular input image with 3D virtual hair.

公开(公告)号：US12094073B2

公开(公告)日：2024-09-17

申请号：US17814391

申请日：2022-07-22

Applicant: Snap Inc.

Inventor： Menglei Chai ,  Sergey Tulyakov ,  Jian Ren ,  Hsin-Ying Lee ,  Kyle Olszewski ,  Zeng Huang ,  Zezhou Cheng

IPC: G06T19/20 ,  G06T9/00 ,  G06T11/00 ,  G06T17/00

CPC classification number: G06T19/20 ,  G06T17/00 ,  G06T2219/2012 ,  G06T2219/2021

Abstract: Systems, computer readable media, and methods herein describe an editing system where a three-dimensional (3D) object can be edited by editing a 2D sketch or 2D RGB views of the 3D object. The editing system uses multi-modal (MM) variational auto-decoders (VADs)(MM-VADs) that are trained with a shared latent space that enables editing 3D objects by editing 2D sketches of the 3D objects. The system determines a latent code that corresponds to an edited or sketched 2D sketch. The latent code is then used to generate a 3D object using the MM-VADs with the latent code as input. The latent space is divided into a latent space for shapes and a latent space for colors. The MM-VADs are trained with variational auto-encoders (VAE) and a ground truth.

公开(公告)号：US11995781B2

公开(公告)日：2024-05-28

申请号：US17987285

申请日：2022-11-15

Applicant: Snap Inc.

Inventor： Artem Bondich ,  Menglei Chai ,  Oleksandr Pyshchenko ,  Jian Ren ,  Sergey Tulyakov

IPC: G06T19/00 ,  G06F18/213 ,  G06F18/214 ,  G06N3/045 ,  G06N3/08 ,  G06T7/246 ,  G06T17/00 ,  G06T19/20

CPC classification number: G06T19/006 ,  G06F18/213 ,  G06F18/214 ,  G06N3/045 ,  G06N3/08 ,  G06T7/251 ,  G06T17/00 ,  G06T19/20 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2215/16 ,  G06T2219/024 ,  G06T2219/2024

Abstract: A messaging system performs neural network hair rendering for images provided by users of the messaging system. A method of neural network hair rendering includes processing a three-dimensional (3D) model of fake hair and a first real hair image depicting a first person to generate a fake hair structure, and encoding, using a fake hair encoder neural subnetwork, the fake hair structure to generate a coded fake hair structure. The method further includes processing, using a cross-domain structure embedding neural subnetwork, the coded fake hair structure to generate a fake and real hair structure, and encoding, using an appearance encoder neural subnetwork, a second real hair image depicting a second person having a second head to generate an appearance map. The method further includes processing, using a real appearance renderer neural subnetwork, the appearance map and the fake and real hair structure to generate a synthesized real image.

公开(公告)号：US11836835B2

公开(公告)日：2023-12-05

申请号：US17364218

申请日：2021-06-30

Applicant: Snap Inc.

Inventor： Menglei Chai ,  Jian Ren ,  Aliaksandr Siarohin ,  Sergey Tulyakov ,  Oliver Woodford

IPC: G06T13/00 ,  G06N3/08 ,  G06T7/20 ,  G06T11/00 ,  G06N3/045

CPC classification number: G06T13/00 ,  G06N3/045 ,  G06N3/08 ,  G06T7/20 ,  G06T11/001 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: Systems and methods herein describe novel motion representations for animating articulated objects consisting of distinct parts. The described systems and method access source image data, identify driving image data to modify image feature data in the source image sequence data, generate, using an image transformation neural network, modified source image data comprising a plurality of modified source images depicting modified versions of the image feature data, the image transformation neural network being trained to identify, for each image in the source image data, a driving image from the driving image data, the identified driving image being implemented by the image transformation neural network to modify a corresponding source image in the source image data using motion estimation differences between the identified driving image and the corresponding source image, and stores the modified source image data.

公开(公告)号：US20230343033A1

公开(公告)日：2023-10-26

申请号：US18216327

申请日：2023-06-29

Applicant: Snap Inc.

Inventor： Chen Cao ,  Menglei Chai ,  Linjie Luo ,  Soumyadip Sengupta

IPC: G06T17/30 ,  G06V20/64 ,  G06V40/16

CPC classification number: G06T17/30 ,  G06V20/647 ,  G06V40/165 ,  G06V40/171

Abstract: A shape generation system can generate a three-dimensional (3D) model of an object from a two-dimensional (2D) image of the object by projecting vectors onto light cones created from the 2D image. The projected vectors can be used to more accurately create the 3D model of the object based on image element (e.g., pixel) values of the image.

公开(公告)号：US20220101104A1

公开(公告)日：2022-03-31

申请号：US17491226

申请日：2021-09-30

Applicant: Snap Inc.

Inventor： Menglei Chai ,  Kyle Olszewski ,  Jian Ren ,  Yu Tian ,  Sergey Tulyakov

IPC: G06N3/04 ,  G06K9/62 ,  G06N3/08 ,  G06T7/20

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN, by performing operations comprising identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthesized video based on the primary GAN and the input data.