The present invention relates to a contact plug of a semiconductor device and a method of forming the same. The method includes forming an insulating layer over a semiconductor substrate, forming contact holes in the insulating layer, forming a first conductive layer over the insulating layer including the contact holes, etching the first conductive layer so that the first conductive layer remains at lower portions of the contact holes, wherein the insulating layer is also etched in order to widen upper widths of the contact holes, and forming a second conductive layer over the first conductive layer of the contact holes, thus forming the contact plugs.