A method of processing moving picture and an apparatus thereof are disclosed. A method of processing a moving picture comprising: calculating a color distribution vector for each of plural sub-frames which are generated by dividing a frame of the moving picture; generating a first order differential of the color distribution vector based on the color distribution vector; generating a second order differential of the color distribution vector based on the first order differential of the color distribution vector; and generating a feature vector of the frame based on the color distribution vector, the first order differential of the color distribution vector and the second order differential of the color distribution vector, is provided. The moving picture processing method can provide an efficient mean to determine commonality between moving pictures by extracting a feature from a frame of the moving pictures.