An apparatus of decoding an image is discussed. The apparatus can include an intra prediction unit for generating a prediction block using an intra prediction mode of a current prediction unit; an inverse scanning unit for inversely scanning significant flags, coefficient signs and coefficient levels according to an inverse scan pattern which is selected based on the intra prediction mode and a size of a transform unit to generate a quantized block; an inverse quantization unit for inversely quantizing the quantized block using a quantization parameter and a quantization matrix to generate a transform block; an inverse transform unit for inversely transform the transform block to generate a residual block; and an adder for adding the prediction block and the residual block to generate a reconstructed block.