The invention discloses a part causal relationship-based video behavior identification method, and mainly solves the problem of low behavior identification accuracy because an existing method ignoresa causal relationship among behavior parts. The method is implemented by the steps of 1) extracting a dense track of a video, and obtaining 3D coordinates and track bottom layer features of track points; 2) by utilizing the 3D coordinates of the track points, performing track clustering to obtain the behavior parts; 3) calculating a ratio of the causal relationship among the parts, training a classifier corresponding to the paired parts, obtaining a judgment score, constructing a descriptor of the causal relationship among the parts, and taking the descriptor as a middle layer semantic representation; and 4) performing PCA dimension reduction on the middle layer semantic representation, and performing behavior identification by utilizing an SVM classifier. According to the method, richer middle layer semantic representations can be obtained, so that the behavior identification accuracy is further improved; and the method can be used for man-machine interaction.