A method includes forming a dummy gate stack over a semiconductor region, forming a gate spacer on a sidewall of the dummy gate stack, removing the dummy gate stack to form an opening, forming a replacement gate stack in the opening, recessing the replacement gate stack to form a recess, filling the recess with a conductive material, and performing a planarization to remove excess portions of the conductive material over the gate spacer. A remaining portion of the conductive material forms a gate contact plug. A top portion of the gate contact plug is at a same level as a top portion of the first gate spacer.