A memory mapping apparatus and a method in a video decoder/encoder utilize a dividing unit, a storage unit and a memory allocation unit. The dividing unit divides picture data into macroblocks and divides each of the macroblocks, which are units of encoding, into partitions having a predetermined size. A storage unit comprises at least one memory bank which is independently accessible. A memory allocation unit allocates adjacent partitions to different memory banks. Accordingly, a total number of cycles necessary to read and write the picture data is reduced.