This disclosure is directed to validating a calibration of and/or calibrating sensors using semantic segmentation information about an environment. For example, the semantic segmentation information can identify bounds of objects, such as invariant objects, in the environment. Techniques described herein may determine sensor data associated with the invariant objects and compare that data to a feature known from the invariant object. Misalignment of sensor data with the known feature can be indicative of a calibration error. In some implementations, the calibration error can be determined as a distance between the sensor data and a line or plane representing a portion of the invariant object.