Systems and method for refining a depth map of a scene based upon a captured image of the scene. A captured depth map of the scene may contain outage areas or other areas of low confidence. The depth map may be aligned with a color image of the scene, and the depth values of the depth map may be adjusted based upon corresponding color values of the color image. An amount of refinement for each depth value of the aligned depth map is based upon the confidence value of the depth value and a smoothing function based upon a corresponding location of the depth value on the color image.