Detection of motion deviation in a video sequence is provided. Change grids each comprise elements generated by storing in each element of the change grid an indication of whether there is change between corresponding elements of at least two images. A current direction grid is generated from a pair of change grids by searching for movement of a corresponding segment identified in each change grid, the movement occurring between the locations of the segment in each of the pair of change grids and, storing in elements of the current direction grid a vector corresponding to the movement of the segment. A vector stored in an element of the current direction grid is compared with a reference vector. It is determined whether there is motion deviation in the video sequence in accordance with the comparison.