The present invention provides a method of decoding a video signal using graph-based transform comprising the steps: extracting prediction unit partition information of a current coding unit from the video signal; obtaining a graph-based transform kernel from predetermined table information based on the prediction unit partition information; and performing an inverse-transform of a transform unit using the graph-based transform kernel, wherein the graph-based transform kernel corresponds to at least one of the prediction unit partition information and an edge weight, and the edge weight is a predetermined value representing a correlation between pixels.