A memory accessing apparatus and a method thereof are provided, which are suitable for methods and apparatuses that access memory data in blocks, such as a video decoder. An advanced memory mapping method is adopted to generate the column address and row address of a data word in a memory based on the logic address of the data word in a video frame. Macroblocks are organized into clusters, macroblocks of the same cluster are stored in the same row of the memory, and the orders in which the video decoder reads and writes macroblocks are rearranged so as to reduce the number of row changes in memory access and improve the throughput of memory access. Furthermore, memory accesses between the video decoder and the display controller can be coordinated.