A method of fabricating a semiconductor device is provided. A substrate is first provided, and then several IO devices and several core devices are formed on the substrate, wherein those IO devices include IO PMOS and IO NMOS, and those core devices include core PMOS and core NMOS. Thereafter, a buffer layer is formed on the substrate, and then the buffer layer except a surface of the IO PMOS is removed in order to reduce the negative bias temperature instability (NBTI) of the IO PMOS. Afterwards, a tensile contact etching stop layer (CESL) is formed on the IO NMOS and the core NMOS, and a compressive CESL is formed the core PMOS.