A method of manufacturing a structure includes forming an in-process alternating stack including insulating layers and spacer material layers over a substrate, forming two sets of stepped surfaces by dividing the in-process alternating stack into a first alternating stack and a second alternating stack, the first alternating stack having first stepped surfaces and the second alternating stack having second stepped surfaces, forming at least one memory stack structure through the first alternating stack, each of the at least one memory stack structure including charge storage regions, a tunneling dielectric, and a semiconductor channel, replacing portions of the insulating layers in the first alternating stack with electrically conductive layers while leaving intact portions of the insulating layers in the second alternating stack, and forming a contact via structure through the second alternating stack to contact a peripheral semiconductor device under the second stack.