After a groove is formed in an insulating layer formed on a semiconductor substrate, a barrier metal layer is formed on the insulating layer by an ALD process so as to cover the side walls and bottom of the groove, and an impurity layer is formed in or on the surface of the barrier metal layer by an ion implantation process or by an ALD process. Thereafter, the barrier metal layer and the impurity layer are alloyed, and then an inlaid interconnect layer, which is composed of a Cu seed layer and a Cu plating layer, is formed in the groove. Then, an impurity element in the alloyed barrier metal layer is thermally diffused into the inlaid interconnect layer.