Systems and methods for verifying an identity of a party to a video call are disclosed. An example method may be performed by one or more processors of a verification system and include storing first facial encodings, each of the first facial encodings associated with one or more images of a corresponding registered user of a plurality of registered users associated with the verification system, receiving, during a video call associated with a specified registered user of the plurality of registered users, a second facial encoding associated with one or more images of a person participating in the video call, the second facial encoding received from a device with which the person uses to participate in the video call, identifying, among the first facial encodings, a first facial encoding corresponding to the specified registered user, and determining whether the person participating in the video call is the specified registered user.