The encoding device includes: a predictor configured to generate, for each component, a prediction block corresponding to an encoding-target block; a residual generator configured to generate, for each component, a prediction residual representing a difference between the encoding-target block and the prediction block; a mode selector configured to select one mode either an individual encoding mode performing a transform process and a quantization process on a prediction residual of the first component and a prediction residual of the second component for each single component, or a joint encoding mode performing a transform process and a quantization process on a joint prediction residual generated from the prediction residual of the first component and the prediction residual of the second component; a quantization controller configured to determine a quantization matrix to be applied in the quantization process based on the mode selected by the mode selector.