A semiconductor device includes a device isolation structure, a recess channel structure, a first lower gate conductive layer conformal to the recess channel structure and defining a recess, a holding layer over the first lower gate conductive layer to fill the recess defined by the first lower gate conductive layer, and a second lower gate conductive layer over the first lower gate conductive layer and the holding layer. The holding layer is configured to hold a shift of the seam occurring in the recess channel structure.