Methods and systems for detecting motion using video data from a smart camera device. In some embodiments, and by non-limiting example, this disclosure is directed to detecting motion in the compressed video domain. One aspect is a method comprising receiving compressed video data, extracting macroblocks and motion vectors for a plurality of frames in the compressed video data, identifying frame-level features for each of the plurality of frames based on the macroblocks and the motion vectors, calculating similarity features for each of the identified frame-level features based on the frame-level features identified in consecutive frames, and predicting motion for each of the plurality of frames by providing the frame-level features and the similarity features into a model trained to detect motion.