Provided is an apparatus for decoding an image that adaptively filters reference pixels according to an intra prediction mode and a size of a prediction block, and generates the prediction block according to the intra prediction mode. When the reference pixels for a second directional intra prediction mode are filtered, the reference pixels for a first directional intra prediction mode, that is closer to an intra prediction mode having a direction of 45° with reference to a horizontal mode than the second directional intra prediction mode is, are also filtered. The first and the second intra prediction modes exist between the horizontal mode and the intra prediction mode having the direction of 45° with reference to the horizontal mode. The number of intra prediction modes in which the filter is applied increases as the size of the prediction block increases.