A method for processing video signals on the basis of inter prediction includes the steps of: deriving a motion vector predictor by using motion information of a neighboring block of the current block; parsing layer information for indicating the current layer, to which a motion vector difference used for the inter prediction of the current block belongs, within a predefined layer structure in which a combination of motion vector differences between one or more horizontal and vertical components is classified into a plurality of layers; parsing index information for indicating a specific combination in the current layer; deriving a motion vector difference of the current block by using the layer information and the index information; and deriving a motion vector of the current block by adding the motion vector difference to the motion vector predictor.