A semiconductor memory device includes: a lower pillar protruding from a substrate in a vertical direction and extending in a first direction by a trench formed in the first direction; an upper pillar protruding on the lower pillar in a second direction perpendicular to the first direction; a buried bit line junction region disposed on one sidewall of the lower pillar; a buried bit line contacting the buried bit line junction region and filling a portion of the trench; an etch stop film disposed on an exposed surface of the buried bit line; a first interlayer dielectric film recessed to expose a portion of an outer side of at least the upper pillar disposed on the etch stop film; a second interlayer dielectric film disposed on the first interlayer dielectric film; and a gate surrounding the exposed outer side of the upper pillar and crossing the buried bit line.