First memory openings are formed through a first alternating stack of first insulating layers and first spacer material layers. Each first memory opening is filled with a first memory film, a sacrificial dielectric liner, and a first-tier opening fill material portion. Second memory openings are formed through a second alternating stack of second insulating layers and second spacer material layers. A second memory film is formed in each second memory opening. The first-tier opening fill material portions are removed selective to the sacrificial dielectric liners. The sacrificial dielectric liners are removed selective to the second memory films and the first memory films. A vertical semiconductor channel can be formed on each vertical stack of a first memory film and a second memory film.