A method of fabricating a semiconductor device is disclosed. The method includes the steps of forming recesses in a semiconductor substrate; epitaxial growing a first SiGe seed layer with constant Ge content in the recesses; epitaxial growing a second SiGe layer with a constant Ge content higher than the Ge content of first SiGe seed layer on the first SiGe seed layer; epitaxial growing a third SiGe layer with a constant Ge content lower than the Ge content of the second SiGe layer; and forming a cap layer on the third SiGe layer.