A three-dimensional memory device includes a first-tier alternating stack of first insulating layers and first electrically conductive layers located over a substrate, an etch stop material layer located over the first-tier alternating stack, a second-tier alternating stack of second insulating layers and second electrically conductive layers located over the etch stop material layer, inter-tier memory openings vertically extending through the second-tier alternating stack, the etch stop material layer, and the first-tier alternating stack, and memory opening fill structures each including a memory film and a vertical semiconductor channel located in the inter-tier memory openings. The material of the etch stop material layer is different from materials of the first insulating layers, the second insulating layers, the first electrically conductive layers, and the second electrically conductive layers.