A semiconductor memory device includes a connecting member including a semiconductor material, a first electrode film, a first insulating film, a stacked body and three or more semiconductor pillars. The stacked body includes second electrode films and second insulating films that alternately stacked. The semiconductor pillars are arrayed along two or more directions, extend in a stacking direction, pierce through the stacked body and the first insulating film, and are connected to the connecting member. The device includes a third insulating film provided between the semiconductor pillars and the stacked body and between the connecting member and the first electrode film. A charge storage layer is provided at least between one of the second electrode films and the third insulating film.