Provided herein may be a semiconductor memory device and a method of manufacturing the same. The semiconductor memory device may include a stacked body including alternately stacked interlayer insulating layers and conductive patterns, and channel structures penetrating the stacked body. Each of the channel structures may include a channel layer vertically extending up to the height of the upper portion of at least one upper conductive pattern disposed uppermost, among the conductive patterns, a memory layer surrounding the channel layer and extending from the lower interlayer insulating layer to the height of the middle portion of the upper conductive pattern, and a doped semiconductor pattern disposed above the channel layer and the memory layer.