A method for fabricating a semiconductor device includes forming an NMOS region and a PMOS region in a substrate, forming a first stack layer including a first gate dielectric layer and a first work function layer that is disposed over the first gate dielectric layer and contains aluminum, over the PMOS region of the substrate, forming a second stack layer including a second gate dielectric layer, a threshold voltage modulation layer that is disposed over the second gate dielectric layer and contains lanthanum, and a second work function layer that is disposed over the threshold voltage modulation layer, over the NMOS region of the substrate, and annealing the first stack layer and the second stack layer, thereby forming a first dipole-interface by diffusion of the aluminum in the first gate dielectric layer and a second dipole-interface by diffusion of the lanthanum in the second gate dielectric layer, respectively.