Provided is a video decoding method for reconstructing an image, the video decoding method including: obtaining reference image data from a bitstream; determining an attribute of the reference image data as a long-term reference attribute or a short-term reference attribute, according to a frequency of referring to the reference image data by image data to be decoded; storing the reference image data in a memory by using the attribute of the reference image data; and decoding an image by using the reference image data stored in the memory.