公开(公告)号：US11436498B2

公开(公告)日：2022-09-06

申请号：US16896774

申请日：2020-06-09

Applicant: Toyota Research Institute, Inc.

Inventor： Adrien David Gaidon ,  Jie Li ,  Vitor Guizilini

IPC: G06N3/08 ,  G06T7/55 ,  H04N13/204 ,  H04N13/00

Abstract: A neural architecture search system for generating a neural network includes one or more processors and a memory. The memory includes a generator module, a self-supervised training module, and an output module. The modules cause the one or more processors to generate a candidate neural network by a controller neural network, obtain training data, generate an output by the candidate neural network performing a specific task using the training data as an input, determine a loss value using a loss function that considers the output of the candidate neural network and at least a portion of the training data, adjust the one or more model weights of the controller neural network based on the loss value, and output the candidate neural network. The candidate neural network may be derived from the controller neural network and one or more model weights of the controller neural network.

公开(公告)号：US20220148204A1

公开(公告)日：2022-05-12

申请号：US17176336

申请日：2021-02-16

Applicant: Toyota Research Institute, Inc.

Inventor： Vitor Guizilini ,  Rares A. Ambrus ,  Adrien David Gaidon

IPC: G06T7/50 ,  G06T9/00 ,  G06K9/46 ,  G06K9/00 ,  B60W60/00 ,  G01S17/08 ,  G01S17/89 ,  G01B11/22

Abstract: System, methods, and other embodiments described herein relate to determining depths of a scene from a monocular image. In one embodiment, a method includes generating depth features from sensor data according to whether the sensor data includes sparse depth data. The method includes selectively injecting the depth features into a depth model. The method includes generating a depth map from at least a monocular image using the depth model that is guided by the depth features when injected. The method includes providing the depth map as depth estimates of objects represented in the monocular image.

公开(公告)号：US11238601B2

公开(公告)日：2022-02-01

申请号：US16899425

申请日：2020-06-11

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Vitor Guizilini ,  Adrien David Gaidon

IPC: G06T7/00 ,  G06T7/50 ,  G06T3/40

Abstract: A method for estimating depth is presented. The method includes generating, at each decoding layer of a neural network, decoded features of an input image. The method also includes upsampling, at each decoding layer, the decoded features to a resolution of a final output of the neural network. The method still further includes concatenating, at each decoding layer, the upsampled decoded features with features generated at a convolution layer of the neural network. The method additionally includes sequentially receiving the concatenated upsampled decoded features at a long-short term memory (LSTM) module of the neural network from each decoding layer. The method still further includes generating, at the LSTM module, a depth estimate of the input image after receiving the concatenated upsampled inverse depth estimate from a final layer of a decoder of the neural network. The method also includes controlling an action of an agent based on the depth estimate.

公开(公告)号：US20210383240A1

公开(公告)日：2021-12-09

申请号：US16896774

申请日：2020-06-09

Applicant: Toyota Research Institute, Inc.

Inventor： Adrien David Gaidon ,  Jie Li ,  Vitor Guizilini

IPC: G06N3/08 ,  G06T7/55 ,  H04N13/204

Abstract: A neural architecture search system for generating a neural network includes one or more processors and a memory. The memory includes a generator module, a self-supervised training module, and an output module. The modules cause the one or more processors to generate a candidate neural network by a controller neural network, obtain training data, generate an output by the candidate neural network performing a specific task using the training data as an input, determine a loss value using a loss function that considers the output of the candidate neural network and at least a portion of the training data, adjust the one or more model weights of the controller neural network based on the loss value, and output the candidate neural network. The candidate neural network may be derived from the controller neural network and one or more model weights of the controller neural network.

公开(公告)号：US20210358296A1

公开(公告)日：2021-11-18

申请号：US16876699

申请日：2020-05-18

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Kuan-Hui LEE ,  Matthew T. Kliemann ,  Adrien David Gaidon

IPC: G08G1/01 ,  G06T7/246 ,  B60W60/00 ,  G06K9/62

Abstract: Systems and methods determining velocity of an object associated with a three-dimensional (3D) scene may include: a LIDAR system generating two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; a pillar feature network encoding data of the point cloud data to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and a feature pyramid network encoding the pillar features and performing a 2D optical flow estimation to estimate the velocity of the object.

公开(公告)号：US11138751B2

公开(公告)日：2021-10-05

申请号：US16689501

申请日：2019-11-20

Applicant: Toyota Research Institute, Inc.

Inventor： Vitor Guizilini ,  Sudeep Pillai ,  Rares A. Ambrus ,  Jie Li ,  Adrien David Gaidon

IPC: G06T7/55 ,  G06T7/521 ,  G06T7/70 ,  G06T7/20 ,  G06N20/00 ,  G06K9/62 ,  G06N5/04 ,  G06K9/00 ,  G01S7/48

Abstract: System, methods, and other embodiments described herein relate to training a depth model for monocular depth estimation. In one embodiment, a method includes generating, as part of training the depth model according to a supervised training stage, a depth map from a first image of a pair of training images using the depth model. The pair of training images are separate frames depicting a scene from a monocular video. The method includes generating a transformation from the first image and a second image of the pair using a pose model. The method includes computing a supervised loss based, at least in part, on reprojecting the depth map and training depth data onto an image space of the second image according to at least the transformation. The method includes updating the depth model and the pose model according to at least the supervised loss.

公开(公告)号：US20210090277A1

公开(公告)日：2021-03-25

申请号：US16828196

申请日：2020-03-24

Applicant: Toyota Research Institute, Inc.

Inventor： Vitor Guizilini ,  Rares A. Ambrus ,  Rui Hou ,  Jie Li ,  Adrien David Gaidon

IPC: G06T7/55 ,  G06T7/20 ,  G06N3/08 ,  G06N3/04

Abstract: System, methods, and other embodiments described herein relate to self-supervised training for monocular depth estimation. In one embodiment, a method includes filtering disfavored images from first training data to produce second training data that is a subsampled version of the first training data. The disfavored images correspond with anomaly maps within a set of depth maps. The first depth model is trained according to the first training data and generates the depth maps from the first training data after initially being trained with the first training data. The method includes training a second depth model according to a self-supervised training process using the second training data. The method includes providing the second depth model to infer distances from monocular images.

公开(公告)号：US12288348B2

公开(公告)日：2025-04-29

申请号：US18344700

申请日：2023-06-29

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Jiexiong Tang ,  Rares Andrei Ambrus ,  Vitor Guizilini ,  Adrien David Gaidon

IPC: G06T7/55 ,  G06T3/18 ,  G06T3/4007 ,  G06T3/4046 ,  G06T7/70 ,  G06T19/00 ,  G06V20/56

Abstract: A method for depth estimation performed by a depth estimation system associated with an agent includes determining a first depth of a first image and a second depth of a second image, the first image and the second image being captured by a sensor associated with the agent. The method also includes generating a first 3D image of the first image based on the first depth, a first pose associated with the sensor, and the second image. The method further includes generating a warped depth image based on transforming the first depth in accordance with the first pose. The method also includes updating the first pose based on a second pose associated with the warped depth image and the second depth, and updating the first 3D image based on the updated first pose. The method further includes controlling an action of the agent based on the updated first 3D image.

公开(公告)号：US20240354973A1

公开(公告)日：2024-10-24

申请号：US18465381

申请日：2023-09-12

Applicant: Toyota Research Institute, Inc.

Inventor： Vitor Campagnolo Guizilini ,  Igor Vasiljevic ,  Dian Chen ,  Adrien David Gaidon ,  Rares A. Ambrus

IPC: G06T7/50 ,  G06T3/40 ,  G06T7/60 ,  G06T15/10

CPC classification number: G06T7/50 ,  G06T3/40 ,  G06T7/60 ,  G06T15/10 ,  G06T2207/20081 ,  G06T2207/30252

Abstract: Systems, methods, and other embodiments described herein relate to augmenting image embeddings using derived geometries for estimating scaled depth. In one embodiment, a method includes generating a geometric viewing vector using pixel coordinates and intrinsic parameters about a camera for an image captured about a scene. The method also includes deriving geometric embeddings from the geometric viewing vector associated with the image for the camera. The method also includes computing a representation by augmenting image embeddings with the geometric embeddings, the image embeddings associated with visual characteristics about the image. The method also includes estimating a scaled depth of the image from the representation.

公开(公告)号：US12080013B2

公开(公告)日：2024-09-03

申请号：US17368703

申请日：2021-07-06

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Jiexiong Tang ,  Rares Andrei Ambrus ,  Sudeep Pillai ,  Vitor Guizilini ,  Adrien David Gaidon

IPC: G06T7/593 ,  G06T7/536 ,  G06T7/70

CPC classification number: G06T7/596 ,  G06T7/536 ,  G06T7/70 ,  G06T2207/10028 ,  G06T2207/30252

Abstract: A method for estimating depth of a scene includes selecting an image of the scene from a sequence of images of the scene captured via an in-vehicle sensor of a first agent. The method also includes identifying previously captured images of the scene. The method further includes selecting a set of images from the previously captured images based on each image of the set of images satisfying depth criteria. The method still further includes estimating the depth of the scene based on the selected image and the selected set of images.