Since complexity is high in derivation of an enhancement merge candidate which is a scalable merge candidate, a problem may arise in that a processing amount is large when the enhancement merge candidate is used in a case where a prediction unit in which it is necessary to derive many merge candidates is small. In a merge candidate derivation section including an enhancement merge candidate section that derives a merge candidate referring to an inter-layer picture and a base merge candidate derivation section that derives a merge candidate not referring to an inter-layer picture (referring to the temporary picture or the adjacent block of the same layer), the merge candidate derived from the base merge candidate and the merge candidate derived from the enhancement merge candidate are added to a merge candidate list in a case where the size of a prediction unit is greater than a predetermined size. The merge candidate derived from the base merge candidate is added to the merge candidate list in a case where the size of the prediction unit is equal to or less than the predetermined size.