A semiconductor memory device includes: a semiconductor substrate; a first impurity region; a second impurity region; a channel region; a first gate formed on a main surface on a side of the first impurity region; a second gate formed on the main surface on a side of the second impurity region, with a second insulating film being interposed; and a third insulating film formed on a side surface of the first gate. An interface between the third insulating film and the semiconductor substrate directly under the third insulating film is located above an interface between the second insulating film and the main surface of the semiconductor substrate directly under the second insulating film. The total number of steps can thus be reduced, and lower cost is achieved.