A distracted driving analysis system for identifying distracted driving events is provided. The system includes a processor in communication with a memory device, and the processor is programmed to: (i) receive labeled training data, the labeled training data including driving event records (a) each labeled as an actual distracted driving event or a passenger event and (b) including phone usage by a user that occurred within a time period of a driving event, (ii) identify common features of the actual distracted driving events and the passenger events by processing the training data using a supervised machine learning algorithm, (iii) generate a trained model based at least in part upon the identified common features, (iv) process a new driving event, (v) assign the new driving event based at least in part upon features of the new driving event, and/or (vi) determine whether the new driving event is an actual distracted driving event or a passenger event.