This disclosure relates to improved techniques for synthesizing three-dimensional (3D) scenes. The techniques can utilize a neural network architecture to analyze images for detecting objects, classifying scenes and objects, and determining degree of freedom information for objects in the images. These tasks can be performed by, at least in part, using inter-object and object-scene dependency information that captures the spatial correlations and dependencies among objects in the images, as well as the correlations and relationships of objects to scenes associated with the images. 3D scenes corresponding to the images can then be synthesized using the inferences provided by the neural network architecture.