A semiconductor memory device includes a stacked body including insulating layers and gate electrode layers alternately stacked in a direction, a semiconductor layer extending in the direction, and a gate insulating layer provided between the semiconductor layer and the gate electrode layer, and including a first layer, a second layer, and a third layer between the first layer and the second layer. The first layer includes a first insulator, the second layer includes at least one oxide selected from aluminum oxide, yttrium oxide, lanthanum oxide, gadolinium oxide, ytterbium oxide, hafnium oxide, and zirconium oxide, the third layer includes at least one material selected from silicon, germanium, silicon germanium and silicon carbide, and the third layer is positioned between the semiconductor layer and the insulating layer.