The disclosure describes techniques for hierarchical power management of memory of an artificial reality system to reduce power consumption by the memory. An example device may be a peripheral device configured to generate artificial reality content for display or a head-mounted display unit (HMD) configured to output artificial reality content for display. The device includes memory divided into multiple memory blocks configurable to operate in a plurality of power modes. The device also includes memory block controllers controlling memory blocks. Each memory block controller controls which power mode in which the corresponding memory block is to operate, independent of any of the other memory blocks. The device includes a memory power controller configured to configure control registers of the memory block controllers to direct the memory block controllers to select one of the plurality of power modes for the memory blocks when the memory blocks are not being accessed.