An encoded bitstream of entropy encoded video data is received by a video decoder. The encoded bitstream represents syntax elements of a sequence of coding blocks. The sequence of coding blocks is recovered by processing a bin sequences associated with each coding block in a processing pipeline, wherein a defined amount of time is allocated to process each coding block in the processing pipeline. The encoded bitstream is arithmetically decoded to produce each bin sequence. The arithmetic decoder is time-wise decoupled from the processing pipeline by storing a plurality of the bin sequences in a buffer memory.