In-process source-level material layers including a source-level sacrificial layer are formed over a substrate, and an alternating stack of insulating layers and spacer material layers and memory stack structures are formed over the in-process source-level layers. A backside trench is formed through the alternating stack, and a source cavity is formed by removing the source-level sacrificial layer employing an etchant provided through the backside trench. A source contact layer including a doped semiconductor material is formed on vertical semiconductor channels of the memory stack structures within the source cavity. The source contact layer includes an unfilled cavity, which is subsequently filled with a silicon nitride liner, a silicon oxide fill material and a semiconductor cap. A semiconductor oxide structure can be formed by filling voids in the silicon oxide fill material by oxidizing the semiconductor cap into a thermal semiconductor oxide material portion.