In one embodiment, an image is broken up into multiple regions or segments, where each segment may be of arbitrary shape, and a transform (multi-scale or otherwise) is applied on the set of segments. In another embodiment, pattern adaptive prediction is used when predicting the next finer level of the transform pyramid. The pattern adaptive prediction uses the parent grid to determine what geometry of a filter is to be used when predicting the child grid. At the boundaries of the domain, the pattern adaptive prediction can be coupled with the domain adaptive prediction technique.