An apparatus for classifying an action in a video of a sports event including receiving a first video segment and a second video segment in the sports video, wherein the second video segment is the segment after next of the first video segment; receiving a predetermined play state of the first video segment and a different predetermined play state of the second video segment, locating a play object in one of a predetermined set of regions in a field of the sports event in the second segment; determining a role of a player interacting with the play object when the play object is located one of the predetermined set of regions in the second video segment; and classifying the action in the first video segment based on the located play object and the determined role.