According to one embodiment, a magnetic memory device includes an interlayer insulating film, a bottom electrode formed in the interlayer insulating film, a buffer layer formed on the bottom electrode, and a stacked structure formed on the buffer layer and including a first magnetic layer functioning as a magnetic storage layer, wherein a portion of the buffer layer located on a central portion of the bottom electrode is thicker than a portion of the buffer layer located on a peripheral portion of the bottom electrode.