A three-dimensional semiconductor device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, memory stack structures extending through the alternating stack and arranged in at least five rows that extend along a first horizontal direction, contact via structures arranged in a same number of rows as the memory stack structures and overlying the memory stack structures, each of the contact via structures being electrically connected to a semiconductor channel of a respective memory stack structure, bit lines contacting a respective contact via structure and extending along a second horizontal direction that is different from the first horizontal direction, and a pair of wall-shaped via structures extending through the alternating stack and laterally extending along the first horizontal direction.