A method for three-dimensional (3D) scene reconstruction by an agent includes estimating an ego-motion of the agent based on a current image from a sequence of images and a previous image from the sequence of images. Each image in the sequence of images may be a two-dimensional (2D) image. The method also includes estimating a depth of the current image via a depth estimation model comprising a group of encoder layers and a group of decoder layers. The method further includes generating a 3D reconstruction of the current image based on the estimated ego-motion and the estimated depth. The method still further includes controlling an action of the agent based on the 3D reconstruction.