A three-dimensional memory device includes alternating stacks of insulating strips and electrically conductive strips located over a substrate and laterally spaced apart among one another by line trenches which laterally extend along a first horizontal direction and are spaced apart along a second horizontal direction, and memory stack structures arranged in rows extending along the first horizontal direction. Each row of memory stack structures is located on a respective sidewall of the line trenches. Each of the memory stack structures includes a vertical semiconductor channel, a tunneling dielectric contacting the vertical semiconductor channel, a charge storage layer contacting the tunneling dielectric, and a composite blocking dielectric. The composite blocking dielectric includes a first dipole-containing blocking dielectric layer stack, a homogeneous blocking dielectric layer, and a second dipole-containing blocking dielectric layer stack.