A memory film layer is formed in a memory opening through an alternating stack of first material layers and second material layers. A sacrificial material layer is deposited on the memory film layer. Horizontal portions of the sacrificial material layer and the memory film layer at the bottom of the memory opening is removed by an anisotropic etch to expose a substrate underlying the memory opening, while vertical portions of the sacrificial material layer protect vertical portions of the memory film layer. After removal of the sacrificial material layer selective to the memory film, a doped semiconductor material layer can be formed directly on the exposed material in the memory opening and on the memory film as a single material layer to form a semiconductor channel of a memory device.