A memory element with a first electrode, a memory layer, and a second electrode in this order. The memory layer includes a resistance change layer provided on the first electrode side, an ion source layer provided on the second electrode side, an intermediate layer provided between the resistance change layer and the ion source layer, and a barrier layer provided at least either between the ion source layer and the intermediate layer, or between the intermediate layer and the resistance change layer.