An insulating film and another insulating film are formed over a semiconductor substrate in that order to cover first, second, and third gate electrodes. The another insulating film is etched back to form sidewall spacers over side surfaces of the insulating film. Then, the sidewall spacers over the side surfaces of the insulating films corresponding to the sidewalls of the first and second gate electrodes are removed to leave the sidewall spacers over the side surfaces of the insulating film corresponding to the sidewalls of the third gate electrode. Then, the sidewall spacers and the insulating films are etched back, so that the sidewall spacers are formed of the insulating film over the sidewalls of the first, second, and third gate electrodes.