A method includes forming a semiconductor fin, performing a first passivation step on a top surface of the semiconductor fin using a first passivation species, and performing a second passivation step on sidewalls of the semiconductor fin using a second passivation species different from the first passivation species. A gate stack is formed on a middle portion of the semiconductor fin. A source or a drain region is formed on a side of the gate stack, wherein the source or drain region and the gate stack form a Fin Field-Effect Transistor (FinFET).