A semiconductor device includes: a semiconductor layer; an insulating film provided on the semiconductor layer; and a charge storage layer provided on the insulating film. The semiconductor layer has a channel formation region in its surface portion. The insulating film contains silicon, germanium, and oxygen. The charge storage layer is capable of storing charge supplied from the semiconductor layer through the insulating film. A method of manufacturing a semiconductor device includes: forming a silicon oxide film on a surface of a semiconductor layer; introducing germanium into the silicon oxide film; forming an insulating film containing silicon, germanium, and oxygen by heat treatment under oxidizing atmosphere; and forming a charge storage layer on the insulating film, the charge storage layer being capable of storing charge supplied from the semiconductor layer through the insulating layer.