A semiconductor device and a method of forming the same are disclosed. The semiconductor device includes a substrate, and a source region and a drain region formed in the substrate. The semiconductor device further includes an impurity diffusion stop layer formed in a recess of the substrate between the source region and the drain region, wherein the impurity diffusion stop layer covers bottom and sidewalls of the recess. The semiconductor device further includes a channel layer formed over the impurity diffusion stop layer and in the recess, and a gate stack formed over the channel layer.