A method of fabricating a semiconductor device having a three-dimensional stacked structure by stacking semiconductor circuit layers on a support substrate, including the steps of: forming a trench in a semiconductor substrate; filling inside the trench with a conductive material to form a conductive plug; forming an element or circuit in an inside or on a surface of the semiconductor substrate where the conductive plug was formed; covering the surface of the semiconductor substrate where the element or circuit was formed with a second insulating film; and fixing the semiconductor substrate to the support substrate or a remaining one of the semiconductor circuit layers by joining the second insulating film to the support substrate or the remaining one of the semiconductor circuit layers through a wiring structure; selectively removing the semiconductor substrate to expose the first insulating film; and selectively removing the first insulating film.