A memory device includes a silicon-germanium source contact layer, an alternating stack of insulating layers and electrically conductive layers located over the silicon-germanium source contact layer, and a memory stack structure vertically extending through the alternating stack. The memory stack structure comprises a memory film and a vertical semiconductor channel that contacts the memory film. The silicon-germanium source contact layer contacts a cylindrical portion of an outer sidewall of the vertical semiconductor channel. Logic circuits for operating the memory elements may be provided on a substrate within a same semiconductor die, or may be provided in another semiconductor die that is bonded to the semiconductor die containing the memory device.