Methods, systems, and apparatus, including computer program products, are provided for refining quilts of patches representing three dimensional structures. A method is provided that includes generating a first quilt representation of a three-dimensional structure, adding information to the first quilt representation, and generating a second quilt representation based on the first quilt representation. The first quilt representation is based on a model and associated with a first display level of detail. The second quilt representation is associated with a second display level of detail higher than the first display level of detail.