A method includes forming a first and a second dummy gate stack crossing over a semiconductor region, forming an ILD to embed the first and the second dummy gate stacks therein, replacing the first and the second dummy gate stacks with a first and a second replacement gate stack, respectively, performing a first etching process to form a first opening. A portion of the first replacement gate stack and a portion of the second replacement gate stack are removed. The method further includes filling the first opening to form a dielectric isolation region, performing a second etching process to form a second opening, with the ILD being etched, and the dielectric isolation region being exposed to the second opening, forming a contact spacer in the second opening, and filling a contact plug in the second opening. The contact plug is between opposite portions of the contact spacer.