A method and system for generating a virtual representation of a physical scene, including receiving scene data corresponding to the physical scene, processing the scene data to determine scene components and scene priors corresponding to the scene components, generating, by a plurality of neural networks, dense geometric representations based at least in part on the scene priors, where each dense geometric representation corresponds to a scene component in the scene components, generating a virtual model of the physical scene based at least in part on the dense geometric representations, and generating a virtual representation of the physical scene based at least in part on the scene data, the virtual representation being aligned with the virtual model.