A semiconductor device is provided. The semiconductor device comprises a substrate, a stacked gate structure, doped regions and high stress material layers. The stacked gate structure is located on the substrate. The stacked gate structure includes at least a dielectric layer and a gate sequentially disposed over the substrate. The doped regions are disposed in the substrate on each side of the stacked gate structure. The high stress material layers are disposed on the substrate to cover the doped regions. The high stress material layers can increase the mobility of the carriers in the doped regions and hence accelerate the operating speed of the device.