A system and method for motion detection and the use thereof in video coding are disclosed. In one aspect, a method of defining a region of motion within a video frame in a sequence of video frames comprises loading a current video frame and at least one reference video frame from the sequence, the reference video frame being different from the current video frame. The method further comprises applying filtering operations on the current and the reference video frame in order to obtain at least two scales of representation of the current and the reference video frame. The method further comprises determining for each of the scale representations a video-frame like representation of the structural changes between the current and the reference video frame. The method further comprises combining the video-frame like representations of different scales. The method further comprises determining one or more regions of motion from the combination.