A digital signal processing apparatus comprises a main memory, a processing unit, a cache, and a rotate buffer unit. The main memory includes at least R memory banks for storing a plurality of data of digital signal. The cache is coupled between the main memory and the processing unit. The cache includes at least R×R cache units for storing part of data of the main memory to provide to the processing unit. The cache also temporarily stores operation results of the processing unit. The rotate buffer unit is coupled between the main memory and the cache for buffering and rotating the data outputted from each of the memory banks to write to the cache, and the data outputted from part of the cache units to write back to the corresponding memory banks respectively.