A system and method processes visual information including at least one object in motion. The visual information is processed by locating at least one spatial edge of the object, generating a plurality of spatio-temporal gradients for the at least one spatial edge over N frames, and then generating motion blur images from the spatio-temporal gradients. A regression analysis is performed on the motion blur images to determine direction of motion information of the object, and scene activity vectors are then generated for the N frames based on the direction of motion information. An event is detected in the visual information based on the scene activity vectors.