A method for managing a reference picture with a decoder is disclosed. The method for managing a reference picture includes storing a decoded picture including quality base picture and quality enhanced picture in a buffer, marking the quality base picture and the quality enhanced picture as a reference picture, marking the quality base picture as distinct information, managing the quality base picture and the quality enhanced picture using an adaptive memory management method or a sliding window method.