A system and method for scene determination is disclosed. The system comprises a communication interface, an object detector, a temporal pattern module and a scene determination module. The communication interface receives a video including at least one frame. The at least one frame includes information describing a scene. The object detector detects a presence of an object in the at least one frame and generates at least one detection result based at least in part on the detection. The temporal pattern module generates a temporal pattern associated with the object based at least in part on the at least one detection result. The scene determination module determines a type of the scene based at least in part on the temporal pattern.