A user authorization system for authorization management is disclosed. The user authorization system includes a public key infrastructure (PKI) certificate issuing server that issues a PKI certificate including a subscriber's biometric signature. The system also includes a sensing means that recognizes biometric patterns. The system also includes a smart card that stores the subscriber's biometric signature and the PKI certificate and verifies the user. The user authorization system also includes a terminal configured to establish a virtual private network (VPN) between the smart card and the service server in response to a tunneling start signal received from the smart card. The smart card transmits the tunneling start signal to the terminal if the user's biometric pattern matches with the subscriber's biometric signature and transmits authorization information derived from the PKI certificate to the service server through the VPN.