A method of forming a three-dimensional memory device includes forming an alternating stack of insulating layers and sacrificial material layers over a substrate, forming a memory opening extending through the alternating stack, forming a sacrificial memory opening fill structure in the memory opening, replacing the sacrificial material layers with electrically conductive layers, removing the sacrificial memory opening fill structure selective to the electrically conductive layers, and forming a memory opening fill structure the memory opening after replacing the sacrificial material layers with electrically conductive layers and after removing the sacrificial memory opening fill structure. The memory opening fill structure includes a memory film and a vertical semiconductor channel.