Decoding tasks are identified for decoding encoded video. Decoding tasks may include entropy decoding tasks, motion compensation tasks, inverse frequency transform tasks, inverse quantization tasks, intra decoding tasks, loop filtering tasks, or other tasks. Task dependencies are identified for the video decoding tasks. For example, one or more decoding tasks may depend on prior completion of entropy decoding tasks. The decoding tasks are prioritized based at least in part on the task dependencies. For example, a higher priority may be assigned to tasks that must be completed before other tasks that depend on them can begin. Prioritized decoding tasks are selected to be performed by hardware threads. For example, a first hardware thread may perform a first decoding task that does not depend on any uncompleted tasks while a second hardware thread performs a second decoding task that does not depend on any uncompleted tasks.