Bit line diffusion layers are formed in an upper part of a semiconductor substrate with a bit line contact region being interposed between the bit line diffusion layers. A conductive film is formed over the semiconductor substrate, the bit line diffusion layers, and first gate insulating films. Then, control gate electrodes are formed from the conductive film. Thereafter, at least the first gate insulating film in the bit line contact region is removed, and a connection diffusion layer is formed in the bit line contact region so as to connect the bit line diffusion layers located on both sides of the bit line contact region. When forming the control gate electrodes, the conductive film is left so as to extend over the bit line contact region and over the bit line diffusion layers located on both sides of the bit line contact region.