A nonvolatile semiconductor memory device includes: a source region and a drain region formed at a distance from each other in a semiconductor substrate; a tunnel insulating film formed on the semiconductor substrate between the source region and the drain region; a charge storage film formed on the tunnel insulating film; a first alumina layer formed on the charge storage film, and having a first impurity element added thereto, the first impurity element having an octacoordinate ion radius of 63 pm or greater, the first impurity element having a concentration distribution in a layer thickness direction of the first alumina layer that becomes the largest in a region close to the side of the charge storage film; a second alumina layer formed on the first alumina layer, and not having the first impurity element added thereto; and a control gate electrode formed on the second alumina layer.