A nonvolatile semiconductor memory device includes: a semiconductor substrate including a first channel, and a source region and a drain region provided on both sides of the first channel; a first insulating film provided on the first channel; a charge retention layer provided on the first insulating film; a second insulating film provided on the charge retention layer; and a semiconductor layer including a second channel provided on the second insulating film, and a source region and a drain region provided on both sides of the second channel.