A method of fabricating a semiconductor device that includes forming a replacement gate structure on a portion of a semiconductor substrate, wherein source regions and drain regions are formed in opposing sides of the replacement gate structure. A dielectric is formed on the semiconductor substrate having an upper surface that is coplanar with an upper surface of the replacement gate structure. The replacement gate structure is removed to provide an opening to an exposed portion of the semiconductor substrate. A functional gate conductor is epitaxially grown within the opening in direct contact with the exposed portion of the semiconductor substrate. The method is applicable to planar metal oxide semiconductor field effect transistors (MOSFETs) and fin field effect transistors (finFETs).