An autonomous vehicle (AV) performance optimization system can receive vehicle data from human-driven vehicles, and determine a set of performance metrics for determining AV performance in relation to human performance based on the vehicle data. Thereafter, the system can receive AV data from AVs operating or configured for operation throughout a given region. Based on the AV data, the system can determine a performance score for each of the set of performance metrics for the AV.