A video decoding apparatus includes a decoder to decode video encoded data using a predictive picture signal for a video signal, a compression device compressing the decoded picture signal, a first memory storing the decoded picture signal, a second memory storing the compressed picture signal, a decompression device decompressing the compressed picture signal read from the second memory, a selector selecting one of a decoded picture signal read from the first memory and a compressed/decompressed picture signal from the compression device as a reference picture signal according to at least one of a coding type of the video encoded data in picture unit and a prediction mode in block unit, and a motion compensator performing motion compensation on the reference picture signal to generate a predictive picture signal.