An apparatus and method are disclosed for user and moving vehicle detection in which sensor data for a portable device is processed to determine whether the portable device is in a moving vehicle. Following a determination the portable device is in a moving vehicle, the sensor data is to characterize an association between the user and the portable device to determine whether the portable device is connected to the user. If the user is connected to the portable device, it is then determined if the portable device is being held in hand. If the portable device is held in hand, it is then determined if the user is operating the moving vehicle. Output from an image sensor of the portable device may be used in determining if the user is the operator.