An apparatus and method for out-focusing a color image based on a depth image, the method including receiving an input of a depth region of interest (ROI) desired to be in focus for performing out-focusing in the depth image, and applying different blur models to pixels corresponding to the depth ROI, and pixels corresponding to a region, other than the depth ROI, in the color image, thereby performing out-focusing on the depth ROI.