A set of cryptographic keys are synchronized across a set of HSMs that are configured in an HSM cluster. The set of cryptographic keys is maintained in a synchronized state by HSM cluster clients running on client computer systems with corresponding client applications. If the HSM cluster becomes unsynchronized, an HSM cluster client attempts to lock the HSM cluster and reestablish synchronization of the cryptographic keys across the HSM cluster. HSMs within the HSM cluster are able to establish an encrypted communication channel to other HSMs without revealing the contents of their communications to their respective host computer systems. Individual HSMs in the HSM cluster may include features that assist the HSM cluster client in determining whether each HSM is up-to-date, identifying particular keys that are not up-to-date, and copying keys from one HSM to another HSM within the HSM cluster.