公开(公告)号：US20230177849A1

公开(公告)日：2023-06-08

申请号：US17543135

申请日：2021-12-06

Applicant: TOYOTA RESEARCH INSTITUTE, INC. ,  THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY

Inventor： Rares Andrei AMBRUS ,  Or LITANY ,  Vitor GUIZILINI ,  Leonidas GUIBAS ,  Adrien David GAIDON ,  Jie LI

IPC: G06K9/00 ,  G06T7/50

CPC classification number: G06K9/00201 ,  G06T7/50 ,  G06K9/00791 ,  G06T2207/30252 ,  G06T2207/20081

Abstract: A method for 3D object detection is described. The method includes concurrently training a monocular depth network and a 3D object detection network. The method also includes predicting, using a trained monocular depth network, a monocular depth map of a monocular image of a video stream. The method further includes inferring a 3D point cloud of a 3D object within the monocular image according to the predicted monocular depth map. The method also includes predicting 3D bounding boxes from a selection of 3D points from the 3D point cloud of the 3D object based on a selection regression loss.

公开(公告)号：US20220300766A1

公开(公告)日：2022-09-22

申请号：US17377161

申请日：2021-07-15

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Vitor GUIZILINI ,  Rares Andrei AMBRUS ,  Adrien David GAIDON ,  Igor VASILJEVIC ,  Gregory SHAKHNAROVICH

IPC: G06K9/62 ,  G06T3/00 ,  G06T3/40 ,  H04N5/247 ,  G06T7/55 ,  G06K9/72

Abstract: A method for multi-camera self-supervised depth evaluation is described. The method includes training a self-supervised depth estimation model and an ego-motion estimation model according to a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes generating a single-scale correction factor according to a depth map of each camera of the multi-camera rig during a time-step. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the self-supervised depth estimation model and the ego-motion estimation model. The method also includes scaling the 360° point cloud according to the single-scale correction factor to form an aligned 360° point cloud.

公开(公告)号：US20220292837A1

公开(公告)日：2022-09-15

申请号：US17830918

申请日：2022-06-02

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Vitor GUIZILINI ,  Adrien David GAIDON

IPC: G06V20/56 ,  G06T7/50 ,  G06V40/10 ,  G06V10/75 ,  G06K9/62

Abstract: A method for navigating a vehicle through an environment includes assigning a first weight to each pixel associated with a dynamic object and assigning a second weight to each pixel associated with a static object. The method also includes generating a dynamic object depth estimate for the dynamic object and generating a static object depth estimate for the static object, an accuracy of the dynamic object depth estimate being greater than an accuracy of the static object depth estimate. The method still further includes generating a 3D estimate of the environment based on the dynamic object depth estimate and the static object depth estimate. The method also includes controlling an action of the vehicle based on the 3D estimate of the environment.

公开(公告)号：US20220108463A1

公开(公告)日：2022-04-07

申请号：US17555112

申请日：2021-12-17

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Vitor GUIZILINI ,  Adrien David GAIDON

IPC: G06T7/50 ,  G06T3/40

Abstract: A method for using an artificial neural network associated with an agent to estimate depth, includes receiving, at the artificial neural network, an input image captured via a sensor associated with the agent. The method also includes upsampling, at each decoding layer of a plurality of decoding layers of the artificial neural network, decoded features associated with the input image to a resolution associated with a final output of the artificial neural network. The method further includes concatenating, at each decoding layer, the upsampled decoded features with features obtained at a convolution layer associated with a respective decoding layer. The method still further includes estimating, at a recurrent module of the artificial neural network, a depth of the input image based on receiving the concatenated upsampled decoded features from each decoding layer. The method also includes controlling an action of an agent based on the depth estimate.

公开(公告)号：US20220005217A1

公开(公告)日：2022-01-06

申请号：US17368703

申请日：2021-07-06

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Jiexiong TANG ,  Rares Andrei AMBRUS ,  Sudeep PILLAI ,  Vitor GUIZILINI ,  Adrien David GAIDON

IPC: G06T7/593 ,  G06T7/70 ,  G06T7/536

Abstract: A method for estimating depth of a scene includes selecting an image of the scene from a sequence of images of the scene captured via an in-vehicle sensor of a first agent. The method also includes identifying previously captured images of the scene. The method further includes selecting a set of images from the previously captured images based on each image of the set of images satisfying depth criteria. The method still further includes estimating the depth of the scene based on the selected image and the selected set of images.

公开(公告)号：US20210365733A1

公开(公告)日：2021-11-25

申请号：US16879497

申请日：2020-05-20

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Vitor GUIZILINI ,  Adrien David GAIDON

IPC: G06K9/62 ,  G06K9/00 ,  G06N3/08 ,  G01C21/36 ,  B60W60/00

Abstract: A method for image reconstruction and domain transfer through an invertible depth network is described. The method includes training a first invertible depth network model using a first image dataset corresponding to a first geographic region to estimate a first depth map. The method also includes retraining the first invertible depth network model using a second image dataset corresponding to a second geographic region to estimate a second depth map. The method further includes reconstructing, by the first invertible depth network model, a third image dataset based on the second depth map. The method also includes training a second invertible depth network model using the third image dataset corresponding to the first geographic region and the second geographic region to estimate a third depth map.

公开(公告)号：US20210350616A1

公开(公告)日：2021-11-11

申请号：US16869341

申请日：2020-05-07

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Vitor GUIZILINI ,  Adrien David GAIDON

IPC: G06T17/05 ,  G06T7/55 ,  G06T5/00 ,  G06N3/08 ,  G06N3/04

Abstract: A method is presented. The method includes estimating an ego-motion of an agent based on a current image from a sequence of images and at least one previous image from the sequence of images. Each image in the sequence of images may be a two-dimensional (2D) image. The method also includes estimating a depth of the current image based the at least one previous image. The estimated depth accounts for a depth uncertainty measurement in the current image and the at least one previous image. The method further includes generating a three-dimensional (3D) reconstruction of the current image based on the estimated ego-motion and the estimated depth. The method still further includes controlling an action of the agent based on the three-dimensional reconstruction.

公开(公告)号：US20210350222A1

公开(公告)日：2021-11-11

申请号：US16867124

申请日：2020-05-05

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Rares A. AMBRUS ,  Vitor GUIZILINI ,  Sudeep PILLAI ,  Adrien David GAIDON

IPC: G06N3/08 ,  G06K9/62 ,  G06T7/50

Abstract: Systems and methods to improve machine learning by explicitly over-fitting environmental data obtained by an imaging system, such as a monocular camera are disclosed. The system includes training self-supervised depth and pose networks in monocular visual data collected from a certain area over multiple passes. Pose and depth networks may be trained by extracting data from multiple images of a single environment or trajectory, allowing the system to overfit the image data.

公开(公告)号：US20210237774A1

公开(公告)日：2021-08-05

申请号：US17093393

申请日：2020-11-09

Applicant: TOYOTA RESEARCH INSTITUTE, INC.

Inventor： Jiexiong TANG ,  Rares A. AMBRUS ,  Vitor GUIZILINI ,  Sudeep PILLAI ,  Hanme KIM ,  Adrien David GAIDON

IPC: B60W60/00 ,  G06K9/46 ,  G06T7/73 ,  G06N3/08

Abstract: A method for learning depth-aware keypoints and associated descriptors from monocular video for monocular visual odometry is described. The method includes training a keypoint network and a depth network to learn depth-aware keypoints and the associated descriptors. The training is based on a target image and a context image from successive images of the monocular video. The method also includes lifting 2D keypoints from the target image to learn 3D keypoints based on a learned depth map from the depth network. The method further includes estimating a trajectory of an ego-vehicle based on the learned 3D keypoints.