A memory die includes dielectric isolation rails embedded within a substrate semiconductor layer, laterally spaced apart along a first horizontal direction, and each laterally extending along a second horizontal direction that is perpendicular to the first horizontal direction, and alternating stacks of insulating layers and electrically conductive layers located over the substrate semiconductor layer. The alternating stacks are laterally spaced apart along the second horizontal direction by line trenches that laterally extend along the first horizontal direction. Arrays of memory stack structures are provided such that each array of memory stack structures among the arrays of memory stack structures vertically extends through a respective alternating stack. Each of the memory stack structures includes a respective vertical stack of memory elements and a respective vertical semiconductor channel.