Embodiments provide a method and system for monitoring user activity in a vehicle. A first sensor in the vehicle acquiring current activity data regarding the user. A second sensor in the vehicle can acquire environment data regarding an environment surrounding the user. A first set of classifiers can generate key point data regarding the user, which indicates body hand/eye movement points of the user. A second set of classifiers can assign the current activity to the activity group. A storage can store response information regarding a user, which includes mappings between activity groups and corresponding response time duration for the user to regain control of the vehicle. The system can generate a take-over request for the user to regain control of the vehicle based on the mapping and current activity.