The present motion description technique provides a technique for defining a motion description offline. The motion description can then later be extracted from a multimedia representation and adapted to various multimedia-related applications in a manner that not only reduces the processing for motion estimation but also provides high compression performance during an encoding/transcoding process. The motion description technique employs a motion alignment scheme utilizing a hierarchical model to describe motion data of each macroblock in a coarse-to-fine manner. Motion information is obtained for motion vectors of macroblocks for different partition modes. The resulting motion information is compressed based on correlations among spatially neighboring macroblocks and among partition modes to form the offline motion description.