There is provided an image processing system which can remove jaggies not only from binary images but also from gray scale images, color images and images upscaled by an unknown scaler. When the center pixel of the local area clipped from a processing target image is decided to be the jaggy removal target pixel, a base filter computing means 84 applies a plurality of types of predetermined base filters to the local area. A weight calculating means 85 calculates weights with respect to the plurality of types of base filters based on the feature amount calculated by a principal component analyzing means 82. Based on a filtering result of each base filter and a weight with respect to each base filter, a filtering result weight computing means 86 updates a pixel value of the center pixel.