The disclosure relates to assessing operation of two or more cameras. These cameras may be a group of cameras of a perception system of a vehicle having an autonomous driving mode. A first image captured by a first camera and a second image captured by a second camera may be received. A first feature vector for the first image and a second feature vector for the second image may be generated. A similarity score may be determined using the first feature vector and the second feature vector. This similarity score may be used to assess the operation of the two cameras and an appropriate action may be taken.