An apparatus and method for sample adaptive offset to restore intensity shift of processed video data are described. In a video coding system, the video data are subject to various processing such as prediction, transformation, quantization, deblocking, and adaptive loop filtering. Along the processing path in the video coding system, certain characteristics of the processed video data may be altered from the original video data due to the operations applied to video data. For example, the mean value of the processed video may be shifted. Therefore, the pixel intensity shift has to be carefully compensated or restored to alleviate the artifacts. Accordingly a sample adaptive offset scheme is disclosed that can take into consideration of the dynamic characteristics within a frame using a region partition scheme. Furthermore, the sample adaptive offset scheme also supports multiple SAO types that can be tailored to the characteristics of processed video data and achieve better quality.