Collateral etching of a dielectric material around a trench during formation of a substrate contact via structure can be avoided employing an aluminum oxide layer. The aluminum oxide layer functions as an etch stop layer during an anisotropic etch that removes horizontal portions of an insulating material layer to form an insulating spacer. The aluminum oxide layer may be a conformal or a non-conformal material layer, and may, or may not, include a horizontal portion that overlies an alternating stack of insulating layers and electrically conductive layers. Electrical shorts caused by widening of the top portion of the trench can be avoided through use of the aluminum oxide layer. Memory stack structures can extend through the alternating stack to provide a three-dimensional memory stack structure. A source region can be formed underneath the trench, and the substrate contact via structure can be employed as a source contact via structure.