A vehicle operation authorization system includes a server that receives identification information, which identifies a first portable electronic device that is used as an electronic key for a vehicle, from the first portable electronic device, verifies the identification information, and transmits an operation permission signal to the vehicle to permit a vehicle operation in accordance with a request from the first portable electronic device. A key registration unit registers an encryption key, which is provided from the server, to the first portable electronic device. An access authorization unit uses the encryption key, which is registered in the first portable electronic device, to grant access authorization for accessing the server to a second portable electronic device so that the second portable electronic device functions as the electronic key authorized to operate the vehicle.