Methods, systems, and/or apparatuses are described for detecting relevant motion of objects of interest (e.g., persons and vehicles) in surveillance videos. As described herein input data based on a plurality of captured images and/or video is received. The input data may then be pre-processed and used as an input into a convolution network that may, in some instances, have elements that perform both spatial-wise max pooling and temporal-wise max pooling. Based on The convolution network may be used to generate a plurality of prediction results of relevant motion of the objects of interest.