A system for unmanned driving of a vehicle includes an autonomous driving kit with a detachable terminal having a modularized autonomous driving sensor and configured to support the autonomous driving function of the vehicle. The autonomous driving kit is configured to grasp a vehicle system including various controllers and various sensors applied to the vehicle through a vehicle communication unit of the vehicle and to control autonomous driving adapted to multiple vehicle types and options for each vehicle through cooperative control of the vehicle system and the autonomous driving sensor. A control server is configured to recognize the vehicle to which the autonomous driving kit is attached, provide a moving route of the vehicle, collect a moving state and surrounding conditions of the vehicle based on image information of cameras and transmit the information to the autonomous driving kit.