A processor comprising a mode indicator, a plurality of processing cores, and a neural network unit (NNU), comprising a memory array, an array of neural processing units (NPU), cache control logic, and selection logic that selectively couples the plurality of NPUs and the cache control logic to the memory array. When the mode indicator indicates a first mode, the selection logic enables the plurality of NPUs to read neural network weights from the memory array to perform computations using the weights. When the mode indicator indicates a second mode, the selection logic enables the plurality of processing cores to access the memory array through the cache control logic as a cache memory.