An alternating stack of insulating layers and sacrificial material layers is formed over a substrate. Memory stack structures are formed through the alternating stack. Each of the memory stack structures includes respective charge storage elements and a respective vertical semiconductor channel contacting an inner sidewall of the respective charge storage elements. The sacrificial material layers are replaced with electrically conductive layers. A polycrystalline aluminum oxide blocking dielectric layer is provided between each charge storage element and a neighboring one of the electrically conductive layers. The polycrystalline aluminum oxide blocking dielectric layer is formed by: depositing an amorphous aluminum oxide layer, converting the amorphous aluminum oxide layer into an in-process polycrystalline aluminum oxide blocking dielectric layer, and by thinning the in-process polycrystalline aluminum oxide blocking dielectric layer.