Aspects of the disclosure relate to identifying and responding to problem conditions for a fleet of aerial vehicles. This may include receiving at one or more processors of one or more server computing devices sensor feedback from an AV of the fleet. A problem condition may be identified using the sensor feedback. A mitigation response for the problem condition relating to a mission assigned to the aerial vehicle may be determined. The mitigation response may be sent to the AV in order to cause the aerial vehicle to maneuver according to the mitigation response and thereby automatically respond to the problem condition.