A method includes forming an etch stop layer over and contacting a gate electrode of a transistor, forming a sacrificial layer over the etch stop layer, and etching the sacrificial layer, the etch stop layer, and an inter-layer dielectric layer to form an opening. The opening is then filled with a metallic material. The sacrificial layer and excess portions of the metallic material over a top surface of the etch stop layer are removed using a removal step including a CMP process. The remaining portion of the metallic material forms a contact plug.