Provided are a method of and an apparatus for deciding an encoding mode for variable block size motion estimation, which can decide an encoding mode quickly and with less computation during variable block size motion estimation. The method includes searching in a reference frame for a macroblock that is most similar to a current macroblock, selecting a temporary encoding mode candidate group for encoding of the current macroblock from among a plurality of encoding mode candidate groups including at least one encoding mode, using encoding mode information of the searched-for macroblock of the reference frame, selecting the decided temporary encoding mode candidate group or an encoding mode candidate group including an encoding mode using a smaller block than the decided temporary encoding mode candidate group as a final encoding mode candidate group, based on the temporary encoding mode candidate group, the complexity of the current macroblock or sub-blocks obtained by dividing the current macroblock, and the difference between the current macroblock or each of the sub-blocks and the reference frame, and performing motion estimation in encoding mode included in the final encoding mode candidate group and deciding an encoding mode of the current macroblock.