An improved system and method for implementing efficient decoding of scalable video bitstreams is provided. A virtual decoded picture buffer is provided for each lower layer of the scalable video bitstream. The virtual decoded picture buffer stores decoded lower layer pictures for reference. The decoded lower layer pictures used for reference are compiled to create a reference picture list for each layer. The reference picture list generated by the virtual decoded picture buffer is used during a direct prediction process instead of a target reference list to correctly decode a current macroblock.