A vision system of a vehicle includes a camera in the vehicle with a field of view that encompasses at least a hand of a driver of the vehicle. A control includes an image processor operable to process image data captured by the camera. The control, responsive to image data processing, is operable to determine a hand of the driver of the vehicle and, responsive to determination of movement of the driver's hand, is operable to determine if the movement is indicative of a gesture for controlling an accessory of the equipped vehicle. The control learns habitual movements of the driver's hand and, responsive to (i) determination that a detected gesture is not indicative of a learned habitual movement and (ii) determination that the detected gesture is indicative of a gesture for controlling an accessory of the equipped vehicle, the control generates an output for controlling the accessory.