A semiconductor device includes a substrate, a semiconductor column vertically disposed on the substrate, a first contact material layer on the substrate and in contact with a lower portion of the semiconductor column, a first insulating material layer on the first contact material layer and having an upper surface lower than an upper surface of the semiconductor column, a gate dielectric material layer on the first insulating material layer and on a portion of sidewalls of the semiconductor column while exposing an upper portion of the semiconductor column, and a gate stack structure on the gate dielectric material layer and surrounding a portion of the gate dielectric material layer on the sidewalls of the semiconductor column. The gate stack structure includes from inside to outside a P-type work function layer, an N-type work function layer, and a gate.