Disclosed systems and methods automatically assess buildings and structures. A device may receive one or more images of a structure, such as a building or portion of the building, and then label and extract relevant data. The device may then train a system to automatically assess other data describing similar buildings or structures based on the labeled and extracted data. After training, the device may then automatically assess new data, and the assessment results may be sent directly to a client or to an agent for review and/or processing.