A semiconductor device includes a stacked structure with insulating layers and conductive layers that are alternately stacked on each other, a hard mask pattern on the stacked structure, a channel structure penetrating the hard mask pattern and the stacked structure, insulating patterns interposed between the insulating layers and the channel structure, wherein the insulating patterns protrude farther towards the channel structure than a sidewall of the hard mask pattern, and a memory layer interposed between the stacked structure and the channel structure, wherein the memory layer fills a space between the insulating patterns.