An apparatus for decoding residual data based on a bit plane and a method thereof, capable of achieving a significant reduction in data traffic between a memory and a functional module in a parallel decoding system, include a variable length decoding module configured to generate residual data for each macroblock from a bit stream, divide the residual data into groups, and generate a bit plane regarding each of the groups, and a variable length decoding memory configured to store the bit plane generated from the variable length decoding module and store the residual data of the groups according to a value of the bit plane.