Systems and methods for generating programmatic designs of structures in accordance with embodiments of the invention are illustrated. One embodiment includes a system for generating a programmatic description of a design of a structure, including a processing system, and a memory accessible by the processing system storing instructions that when read by the processing system direct the processing system to receive a visual description of a structure, identify features in the visual description of the structure, determine a list of candidate components corresponding to the identified features from the visual description of the structure, resolve parameters for each component in the list of candidate components, generate a programmatic description of the structure based on the list of candidate components of the resolved parameters and perform an error check to determine an amount that the programmatic description differs from the visual description.