A semiconductor memory device comprises a semiconductor substrate; a memory block formed on the semiconductor substrate and including plural stacked cell array layers of cell arrays each comprising a plurality of first lines, a plurality of second lines crossing the plurality of first lines, and memory cells connected at intersections of the first and second lines between both lines; and a plurality of contacts extending in the stack direction of the cell array layers and connecting the first lines in the cell arrays with diffusion regions formed on the semiconductor substrate. A certain one of the cell array layers is smaller in the number of the first lines divided and the number of contacts connected than the cell array layers in a lower layer located closer to the semiconductor substrate than the certain one.