One illustrative device disclosed herein includes a layer of insulating material having an upper surface positioned at a first level and a recessed conductive interconnect structure positioned at least partially within the layer of insulating material, wherein the recessed conductive interconnect structure has a recessed upper surface that is positioned at a second level that is below the first level. In this example, the device also includes a recess defined in the recessed conductive interconnect structure, a memory cell positioned above the recessed conductive interconnect structure and a conductive via plug that is conductively coupled to the recessed conductive interconnect structure and a lower conductive material of the memory cell, wherein at least a portion of the conductive via plug is positioned in the recess defined in the recessed conductive interconnect.