An alternating stack of insulating layers and spacer material layers is formed over a source-level sacrificial layer overlying a substrate. The spacer material layers are formed as, or are subsequently replaced with, electrically conductive layers. Memory stack structures including a respective vertical semiconductor channel and a respective memory film are formed through the alternating stack. A source-level cavity is formed by removing the source-level sacrificial layer. Semiconductor pillar structures may be used to provide mechanical support to the alternating stack during formation of the source-level cavity. A source-level semiconductor material layer can be formed in the source-level cavity. The source-level semiconductor material layer adjoins bottom end portions of the vertical semiconductor channels and laterally surrounds the semiconductor pillar structures. The source-level semiconductor material layer may be electrically isolated from a substrate semiconductor material layer in the substrate by a series connection of two p-n junctions having opposite polarities.