A method, medium, and apparatus for encoding and/or decoding video by generating a scalable bitstream formed with a base layer bitstream and an enhancement layer bitstream in order to provide forward compatibility, the scalable bitstream comprising a base layer bitstream including a base quantized level and a base quantization parameter corresponding to a base bit-depth, and an enhancement layer bitstream including a residue between an extended quantized level and a compensated quantized level that is predicted from the base quantized level, and additional quantization information for refining the difference between an extended bit-depth and the base bit-depth, wherein a process for improving image quality is applied to at least one of the base layer and the enhancement layer.