A memory die includes an alternating stack of insulating layers and electrically conductive layers, memory stack structures extending through the alternating stack, and each of the memory stack structures includes a respective vertical semiconductor channel and a respective memory film, drain regions located at a first end of a respective one of the vertical semiconductor channels, and a source layer having a first surface and a second surface. The first surface is located at a second end of each of the vertical semiconductor channels, and a semiconductor wafer is not located over the second surface of the source layer.