Embodiments of mechanisms for forming a micro-electro mechanical system (MEMS) device are provided. The MEMS device includes a CMOS substrate and a MEMS substrate bonded with the CMOS substrate. The CMOS substrate includes a semiconductor substrate, a first dielectric layer formed over the semiconductor substrate, and a plurality of conductive pads formed in the first dielectric layer. The MEMS substrate includes a semiconductor layer having a movable element and a second dielectric layer formed between the semiconductor layer and the CMOS substrate. The MEMS substrate also includes a closed chamber surrounding the movable element. The MEMS substrate further includes a blocking layer formed between the closed chamber and the first dielectric layer of the CMOS substrate. The blocking layer is configured to block gas, coming from the first dielectric layer, from entering the closed chamber.