The present invention relates to a method and system for creating a strong classifier based on motion patterns wherein the strong classifier may be used to determine an action being performed by a body in motion. When creating the strong classifier, action classification is performed by measuring similarities between features within motion patterns. Embodiments of the present invention may utilize candidate part-based action sets and training samples to train one or more weak classifiers that are then used to create a strong classifier.