A digital signal processor (DSP) co-processor according to a clustered architecture with local memories. Each cluster in the architecture includes multiple sub-clusters, each sub-cluster capable of executing one or two instructions that may be specifically directed to a particular DSP operation. The sub-clusters in each cluster communicate with global memory resources by way of a crossbar switch in the cluster. One or more of the sub-clusters has a dedicated local memory that can be accessed in a random access manner, in a vector access manner, or in a streaming or stack manner. The local memory is arranged as a plurality of banks. In response to certain vector access instructions, the input data may be permuted among the banks prior to a write, or permuted after being read from the banks, according to a permutation pattern stored in a register.