An image decoding device includes a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector. The prediction unit is configured to perform refinement processing of setting a search range based on a reference position specified by the motion vector, specifying a corrected reference position having the smallest predetermined cost from the search range, and correcting the motion vector based on the corrected reference position. When a block size of the prediction block is larger than a predetermined block size, the prediction unit is configured to divide the prediction block into sub-block groups and perform the refinement processing for each sub-block.