A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers, and memory stack structures extending through the alternating stack. Each of the memory stack structures includes a vertical semiconductor channel, a tunneling dielectric layer, a vertical stack of discrete silicon nitride memory elements located at levels of the electrically conductive layers, and a vertical stack of discrete silicon oxide blocking dielectric structures laterally surrounding the vertical stack of discrete silicon nitride memory elements. Each of the silicon oxide blocking dielectric structures includes a silicon oxynitride surface region, and an atomic concentration of nitrogen atoms within the silicon oxynitride surface region decreases with a lateral distance from an interface between the silicon oxynitride surface region and a respective one of the silicon nitride memory elements.