To provide a semiconductor memory device comprising a plurality of silicon pillars arranged in a matrix, whose sidewalls are provided with gate electrodes with gate insulating films interposed between the silicon pillars and the gate electrodes and whose top ends are electrically connected to memory elements, and a bit line and a word line provided between the silicon pillars so as to be orthogonal to each other. The bit line is electrically connected to a bottom end of the silicon pillars on both sides of the bit line in alternate rows, and the word line is electrically connected to a gate electrode formed on a sidewall of the silicon pillars on both sides of the word line in alternate columns.