A method, processing system and processor-readable medium for classifying human behavior based on a sequence of frames of a digital video. A 2D convolutional neural network is used to identify key points on a human body, such as human body joints, visible within each frame. An encoded representation of the key points is created for each video frame. The sequence of encoded representations corresponding to the sequence of frames is processed by a 3D CNN trained to identify human behaviors based on key point positions varying over time.