Techniques of authenticating a first device of a user to a second device are disclosed. The method enables the second device to perform authentication using a biometric template stored on the first device and a biometric measurement. Homomorphic encryption may be used by the first device to encrypt the biometric template and the second device to determine an encrypted similarity metric between the biometric template and the biometric measurement. The second device can also determine an encrypted code using an authentication function and the encrypted similarity metric. The second device sends the encrypted code and the encrypted similarity metric to be decrypted by the first device. The second device can receive a response from the first device, indicating whether a decrypted similarity metric exceeds a threshold; and whether the decrypted code matches a test code. The second device can then authenticate the user based on the response.