A method of fabricating a semiconductor memory device to protect a tunneling insulating layer from etching-damage includes the steps of forming sequentially a tunnel insulating layer, a first conductive layer, a dielectric layer and a second conductive layer on a semiconductor substrate; etching the second conductive layer, the dielectric layer and the first conductive layer to form gate patterns, the first conductive layer remaining on the tunnel insulating layer between the gate patterns to prevent the tunnel insulating layer from being exposed; performing a cleaning process to remove impurities generated in the etching step; performing an ion implanting process to mono-crystallize the first conductive layer remaining on the tunnel insulating layer; and performing an oxidation process to form an oxide layer on top and side walls of the gate patterns and to convert the mono-crystallized first conductive layer into an insulating layer.