A vertically repeating stack of a unit layer stack is formed over a substrate. The unit layer stack includes a sacrificial material layer, a lower silicon oxide material layer, a first silicon oxide material layer, and an upper silicon oxide material layer. A memory opening can be formed through the vertically repeating stack, and a layer stack including a blocking dielectric layer, a memory material layer, a tunneling dielectric, and a semiconductor channel can be formed in the memory opening. The sacrificial material layers are replaced with electrically conductive layers. The first silicon oxide material layer can be removed to form backside recesses. Optionally, portions of the memory material layer can be removed to from discrete charge storage regions. The backside recesses can be filled with a low-k dielectric material and/or can include cavities within a dielectric material to provide reduced coupling between electrically conductive layers.