A scalable video bitstream may have an H.264/AVC compatible base layer and a scalable enhancement layer, where scalability refers to color bit depth. The H.264/AVC scalability extension SVC provides also other types of scalability, e.g. spatial scalability where the number of pixels in BL and EL are different. According to the invention, BL information is upsampled in two logical steps, one being texture upsampling and the other being bit depth upsampling. Texture upsampling is a process that increases the number of pixels, and bit depth upsampling is a process that increases the number of values that each pixel can have, corresponding to the pixels color intensity. The upsampled BL data are used to predict the collocated EL. The BL information is upsampled at the encoder side and in the same manner at the decoder side, wherein the upsampling refers to spatial and bit depth characteristics.