A method of fabricating a semiconductor device includes the following steps. At first, two gate stack layers are formed on a semiconductor substrate, and a material layer covering the gate stack layers is formed on the semiconductor substrate. Subsequently, a part of the material layer is removed to form a sacrificial layer between the gate stack layers, and a spacer at the opposite lateral sides of the gate stack layers. Furthermore, a patterned mask covering the gate stack layers and the spacer and exposing the sacrificial layer is formed, and the sacrificial layer is removed.