A nonvolatile semiconductor memory device includes a semiconductor substrate, a first insulation layer formed on the semiconductor substrate, a charge storage layer formed on the first insulation layer, a second insulation layer formed on the charge storage layer and including a first high dielectric insulating film which has a higher relative permittivity than a silicon nitride film and a second high dielectric insulating film which has a higher relative permittivity than a silicon nitride film, the first and second high dielectric insulating films being structured so that a silicon oxide film is interposed between them, a control electrode formed on the second insulation layer, a first portion formed between the charge storage layer and the second insulation layer and containing silicon and nitrogen, and a second portion containing silicon and oxygen and located between the charge storage layer and the second insulation layer.