The present disclosure describes method to form a semiconductor device having a gate dielectric layer with controlled doping and to form multiple devices with different Vt. The method includes forming a gate dielectric layer on a fin structure, forming a buffer layer on the gate dielectric layer, and forming a dopant source layer including a dopant on the buffer layer. The gate dielectric layer includes an interfacial layer on the fin structure and a high-k dielectric layer on the interfacial layer. The method further includes doping a portion of the high-k dielectric layer adjacent to the interfacial layer with the dopant, removing the dopant source layer and the buffer layer, forming a dopant pulling layer on the gate dielectric layer, and tuning the dopant in the gate dielectric layer by the dopant pulling layer.