A three-dimensional memory device includes an alternating stack of source layers and drain layers located over a substrate, memory openings vertically extending through the alternating stack, vertical word lines located in each one of the memory openings and vertically extending through each of the source layers and the drain layers of the alternating stack, vertical stacks of discrete semiconductor channels located in each one of the memory openings and contacting horizontal surfaces of a respective vertically neighboring pair of a source layer of the source layers and a drain layer of the drain layers, and vertical stacks of discrete memory material portions located in each one of the memory openings and laterally surrounding a respective one of the vertical word lines. Each memory material portion is laterally spaced from a respective one of the semiconductor channels by a respective gate dielectric layer.