A semiconductor device with different gate structure configurations and a method of fabricating the same are disclosed. The semiconductor device includes first and second pair of source/drain regions disposed on a substrate, first and second nanostructured channel regions, and first and second gate structures with effective work function values different from each other. The first and second gate structures include first and second high-K gate dielectric layers, first and second barrier metal layers with thicknesses different from each, first and second work function metal (WFM) oxide layers with thicknesses substantially equal to each other disposed on the first and second barrier metal layers, respectively, a first dipole layer disposed between the first WFM oxide layer and the first barrier metal layer, and a second dipole layer disposed between the second WFM oxide layer and the second barrier metal layer.