A method for reducing memory size and amount of data accessed of a buffered-frame storage device in a video codec is provided. It is applicable to a codec compliant with a video encoding and decoding standard. When a block is decompressed, the block is stored in the buffered-frame storage device using either compression data or reconstructed data based on its encoded mode and the positions of its reference frames, or based on a topological pattern and a pre-determined memory capacity constraint. With this method, not only the memory size of the buffered-frame storage device, but also the computational complexity requirement for decoding compression data of reference frames is reduced. The present invention compromises among the buffered-frame memory size, amount of data accessed, and computational complexity requirement.