To provide a semiconductor device having a memory element, and which is manufactured by a simplified manufacturing process. A method of manufacturing a semiconductor device includes, forming a first insulating film to cover a first semiconductor film and a second semiconductor film; forming a first conductive film and a second conductive film over the first semiconductor film and the second semiconductor film, respectively, with the first insulating film interposed therebetween; forming a second insulating film to cover the first conductive film; forming a third conductive film selectively over the first conductive film which is formed over the first semiconductor film, with the second insulating film interposed therebetween, and doping the first semiconductor film with an impurity element with the third conductive film serving as a mask and doping the second semiconductor film with the impurity element through the second conductive film.