Inverse transforms used in video and image compression/decompression, such as DCT/IDCT used in MPEG-2 and MPEG-4, or the integer transforms used in H.264, are usually calculated with fast algorithms, which only take advantage of the symmetry existing in the transform matrix but ignores the peculiarities in the input data. While these kind of fast algorithms can apply to both forward and inverse transforms, they tend to be inefficient in calculating inverse transforms. In inverse transforms, most of the coefficients become zero after quantization, this invention takes advantage of this fact to further simplify the general fast algorithm and speed up the calculation.