A scalable video compression system (100) having an encoder (120), bit extractor (140), and decoder (160) for efficiently encoding and decoding a scalable embedded bitstream (130) at different video resolution, framerate, and video quality levels is provided. Bits can be extracted in order of refinement layer (136), followed by temporal level (132), followed by spatial layer (134), wherein each bit extracted provides an incremental improvement in video decoding quality. Bit extraction can be truncated at a position in the embedded bitstream corresponding to a maximum refinement layer, a maximum temporal level, and a maximum spatial layer. For a given refinement layer, bits are extracted from all spatial layers in a lower temporal level prior to extracting bits from spatial layers in a higher temporal level for prioritizing coding gain to increase video decoding quality, and prior to moving to a next refinement layer.