A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, a memory opening extending through the alternating stack, a semiconductor pedestal channel portion located at a bottom portion of the memory opening, and a memory stack structure located in the memory opening and contacting a top surface of the pedestal channel portion. The memory stack structure includes a memory film and a vertical semiconductor channel located inside the memory film. A bottommost insulating layer among the insulating layers comprises a first silicon oxide material, and at least some of the insulating layers other than the bottommost insulating layer include a second silicon oxide material having a greater density than the first silicon oxide material.