A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, memory openings vertically extending through the alternating stack, and memory opening fill structures located in the memory openings and including a respective memory-level semiconductor channel and a respective memory film. Drain-select-level gate electrodes overlie the alternating stack. Drain-select-level pillar structures extend through a respective one of the drain-select-level gate electrodes. Each drain-select-level semiconductor channel is electrically connected to an underlying one of the memory-level semiconductor channels. A planar insulating spacer layer having a homogeneous composition throughout directly contacts top surfaces of the memory films and bottom surfaces of the drain-select-level gate electrodes.