A method reconstructs at three-dimensional (3D) real-world scene from a single two-dimensional (2D) image by identifying junctions satisfying geometric constraint of the scene based on intersecting lines, vanishing points, and vanishing lines that are orthogonal to each other. Possible layouts of the scene are generated by sampling the 2D image according to the junctions. Then, an energy function is maximized to select an optimal layout from the possible layouts. The energy function use's a conditional random field (CRF) model to evaluate the possible layouts.