A memory opening is formed through an alternating stack of insulating layers and sacrificial material layers located over a substrate. Annular recesses are formed around the memory opening by laterally recessing the sacrificial material layers with respect to the insulating layers. Annular metal portions are formed over recessed sidewalls of the sacrificial material layers within each of the annular recesses by a selective deposition process. Annular backside blocking dielectrics are formed selectively on inner sidewalls of the annular metal portions employing a layer of a self-assembly material that covers surfaces of the insulating layers and inhibits deposition of a dielectric material thereupon. A memory stack structure is formed in the memory opening, and the sacrificial material layers are replaced with electrically conductive layers. The annular backside blocking dielectrics provide electrical isolation for the annular metal portions, which function as control gate electrodes.