A method and device for processing a video signal are disclosed. More specifically, a method of processing a video signal based on inter prediction may comprise: deriving a motion vector predictor based on motion information of a neighboring block of a current block; deriving a motion vector difference of the current block based on layer information and index information; deriving a motion vector of the current block based on the motion vector predictor and the motion vector difference; generating a prediction block of the current block based on the motion vector of the current block; and generating a reconstructed block of the current block based on the prediction block and a residual block of the current block, wherein the layer information includes at least one syntax element indicating a layer group to which the current layer belongs.