An authentication system for device-to-device (D2D) communication and an authentication method thereof are provided. The authentication system further includes first user equipment (UE) and an authentication server. The authentication server is located within a communication range of the first UE. When the first UE sends a connection request to the authentication server, the authentication server performs a routine authentication procedure on the first UE and provides key generation information to the first UE. The authentication server generates a server key according to the key generation information and a key derivation procedure. The first UE generates an equipment key according to the key generation information and the key derivation procedure to obtain authentication for D2D communication, such that the first user equipment and a second UE obtaining the authentication for D2D communication directly perform the D2D communication without performing the D2D communication through the authentication server.