Embodiments of the present invention are directed to a method and apparatus for block based image compression with multiple non-uniform block encodings. In one embodiment, an image is divided into blocks of pixels. In one embodiment the blocks are four pixels by four pixels, but other block sizes are used in other embodiments. In one embodiment, a block of pixels in the original image is compressed using two different methods to produce a first and second compressed block. Thus, each block in the original image is represented by two, typically different, compressed blocks. In one embodiment, color associated with a pixel is determined by combining the compressed information about the pixel in the first compressed block with information about the pixel in the second compressed block. In another embodiment, global information about the image is combined with the information in the first and second compressed blocks.