A system for recognizing gestures generates a skeletal model from video data of a subject. A defined subset of attributes of the skeletal model are mapped to defined positions of a vector. A gesture is recognized by evaluating a neural network using the vector as input. The neural network, trained using training vectors generated according to the definitions of skeletal model attributes and vector positions, classifies a gesture based on the input vector.