Disclosed is a method of forming a semiconductor device. The method includes providing a precursor having a substrate and gate stacks over the substrate, wherein each of the gate stacks includes an electrode layer, a first hard mask (HM) layer over the electrode layer, and a second HM layer over the first HM layer. The method further includes depositing a dielectric layer over the substrate and the gate stacks and filling spaces between the gate stacks; and performing a first chemical mechanical planarization (CMP) process to partially remove the dielectric layer. The method further includes performing an etching process to remove the second HM layer and to partially remove the dielectric layer, thereby exposing the first HM layer. The method further includes performing a second CMP process to at least partially remove the first HM layer.