The present invention provides an image processing method on the basis of an intra prediction mode and an apparatus therefor. Specifically, a method for processing an image on the basis of an intra prediction mode may comprise the steps of: identifying whether weighted intra prediction is allowed for a current block and whether the weight intra prediction is applied to the current block; identifying whether reference sample filtering is applied to the current block when the weighted intra prediction is not allowed for the current block or when the weighted intra prediction is not applied to the current block; and performing reference sample filtering of reference samples neighboring the current block when the reference sample filtering is applied.