A device includes a first source/drain region of a first conductivity type over a silicon substrate, wherein the first source/drain region is at a higher step of a two-step profile. The first source/drain region includes a germanium-containing region. A second source/drain region is of a second conductivity type opposite the first conductivity type, wherein the second source/drain region is at a lower step of the two-step profile. A gate dielectric includes a vertical portion in contact with a side edge the silicon substrate, and a horizontal portion in contact with a top surface of the silicon substrate at the lower step. The horizontal portion is connected to a lower end of the vertical portion. A gate electrode is directly over the horizontal portion, wherein a sidewall of the gate electrode is in contact with the vertical portion of the gate dielectric.