An encoding apparatus comprises a quantizer quantizing transform coefficients in a block in an image using a first quantization matrix, and an encoder encoding the first quantization matrix, wherein, for encoding the first quantization matrix, if using a first mode, the encoder encodes information indicating that the first quantization matrix is the same as a second quantization matrix having already been encoded, if using a second mode, encodes a difference between an element of the first quantization matrix and an element of the second quantization matrix, and if using a third mode, encodes a difference between an element and another element within the first quantization matrix, wherein a code amount of information representing the first mode is used is less than that of information representing that the second mode is used.