A nonvolatile semiconductor memory device is provided in such a manner that a semiconductor layer is formed over a substrate, a charge accumulating layer is formed over the semiconductor layer with a first insulating layer interposed therebetween, and a gate electrode is provided over the charge accumulating layer with a second insulating layer interposed therebetween. The semiconductor layer includes a channel formation region provided in a region overlapping with the gate electrode, a first impurity region for forming a source region or drain region, which is provided to be adjacent to the channel formation region, and a second impurity region provided to be adjacent to the channel formation region and the first impurity region. A conductivity type of the first impurity region is different from that of the second impurity region.