Disclosed herein are an apparatus and method for processing a textured image. The apparatus includes a filter unit for detecting an edge of an input image and transforming the input image into an image in which a density of the edge is represented; a smoothing unit for removing noise from the transformed image and smoothing the image; a clustering unit for changing a number of regions into which the smoothed image is to be segmented and clustering the smoothed image a preset number of times; and a cluster optimization unit for setting a final number of clusters for the input image by optimizing a number of clusters based on a previously learned ground truth, for selecting an image corresponding to the final number of clusters from results of clustering by which the image is segmented into a different number of regions, and for outputting the selected image.