A structure is formed on a substrate, which includes a stack of alternating layers comprising insulating layers and electrically conductive layers and a plurality of memory stack structures extending through the stack. At least one bridge line structure is formed on top surfaces of a respective subset of the plurality of memory stack structures to provide local lateral electrical connection. At least one dielectric material layer is formed over the at least one bridge line structure and the plurality of memory stack structures. A plurality contact via structures is formed through the dielectric material layer. The plurality of contact via structures includes at least one first contact via structure contacting a top surface of a respective bridge line structure, and second contact via structures contacting a top surface of a respective memory stack structure.