An image decoding apparatus comprises a decoder which decodes data indicating a plurality of values corresponding to a part of a quantization matrix; a generator which derives the plurality of values from the data and generates the matrix; and an inverse quantizing unit which performs the inverse quantizing on an object block using the matrix, wherein, if width or height of the matrix is larger than or equal to a predetermined size, the generating unit generates the matrix by associating a first value among the plurality of values with a first element corresponding to DC component in the matrix, associating a second value with a second element adjacent to the first element, and, for the elements other than the first and second elements, associating each of one or more values with a plurality of elements of the matrix.