A method includes forming a dummy gate stack on a top surface and a sidewall of a middle portion of a semiconductor fin, and forming a spacer layer. The spacer layer includes a first portion on a sidewall of the dummy gate stack, and a second portion on a top surface and a sidewall of a portion of the semiconductor fin. The method further includes performing an implantation on the spacer layer. After the implantation, an anneal is performed. After the anneal, the second portion of the spacer layer is etched, wherein the first portion of the spacer layer remains after the etching. A source/drain region is formed on a side of the semiconductor fin.