A disposable gate structure straddling a semiconductor fin is formed. A source region and a drain region are formed employing the disposable gate structure as an implantation mask. A planarization dielectric layer is formed such that a top surface of the planarization dielectric layer is coplanar with the disposable gate structure. A gate cavity is formed by removing the disposable gate structure. An epitaxial cap layer is deposited on physically exposed semiconductor surfaces of the semiconductor fin by selective epitaxy. A gate dielectric layer is formed on the epitaxial cap layer, and a gate electrode can be formed by filling the gate cavity. The epitaxial cap layer can include a material that reduces the density of interfacial defects at an interface with the gate dielectric layer.