An image coding apparatus includes a block dividing unit configured to divide an input image into a plurality of blocks, a prediction unit configured to perform prediction based on coded pixels to generate prediction errors, a transformation unit configured to perform orthogonal transform to the prediction errors to generate transform coefficients, a quantization matrix generation unit configured to generate quantization matrices that are used to quantize the transform coefficients, a quantization matrix coding unit configured to calculate difference values by scanning the quantization matrices and to code the difference values, a quantization unit configured to generate quantization coefficients by quantizing the generated transform coefficients using the quantization matrices, and a coefficient coding unit configured to code the quantization coefficients, wherein the quantization matrix coding unit is configured to scan coefficients of the quantization matrices in a unidirectional manner to calculate the difference value.