The present invention realizes an electronic signature system with high security level in which abuse of a signature key by a system administrator is prevented. A user sets an authentication information conceived by the user himself to his/her own signature key stored in the tamper resistant device (5) via the terminal device (2). When digitally signing an electronic document, the user transmits his/her own encrypted authentication information to the tamper resistant device (5) through the terminal device (2) and asks for permission to use his/her signature key. The tamper resistant device (5) decodes the inputted authentication information, verifies the decoded authentication information, and allows the digital signing only if the correct authentication information is entered. As a result, the electronic signature system in which only a user having valid use authority for the signature key can digitally sign is built.