Whether a processing target block belongs to a quiescence area is determined based on a motion vector of an adjacent block whose encoding mode is determined to not be the intra-encoding mode and a reference index indicating a reference image of the adjacent block, whereby a reference index indicating a reference image of the processing target block is determined. If all the encoding modes of the adjacent blocks are the intra-encoding mode, an index number assigned to an image temporally closest and in a field different from a field to which the processing target block belongs is selected. If the processing target block belongs to a quiescence area, an index number assigned to an image temporally closest and in a same field as the processing target block is selected. If the processing target block belongs to a moving area, an index number of a reference image temporally closest is selected.