An alternating stack of insulating layers and sacrificial material layers is formed over a substrate, and support pillar structures are formed through the alternating stack. Stepped surfaces are formed by patterning the alternating stack and the support pillar structures. A retro-stepped dielectric material portion is formed over the stepped surfaces. Memory openings and memory opening fill structures are formed through the alternating stack. Electrically conductive layers are formed by replacing at least the sacrificial material layers with at least one electrically conductive material. Contact via structures are formed through the retro-stepped dielectric material portion on the electrically conductive layers. A first support pillar structure is located directly below a first contact via structure.