According to one embodiment, a magnetic memory device includes a stacked structure including a first magnetic layer having a fixed magnetization direction, a nonmagnetic layer provided on the first magnetic layer, and a second magnetic layer provided on the nonmagnetic layer and having a variable magnetization direction, a first insulating layer provided along a side surface of the stacked structure and having an upper end located at a position lower than an upper end of the side surface of the stacked structure, and a second insulating layer covering the first insulating layer and having an upper end located at a position higher than the upper end of the first insulating layer.