A system (28, 32) generates an image registration map. The system (28, 32) includes one or more processors (32) which receive a first image and a second image. Corresponding interest points in the first image and the second image are identified. Corresponding structures in the first and second images are identified and corresponding boundary points are identified on their boundaries. A registration map is generated from pairs of the corresponding interest points and a subset of pairs of the corresponding boundary points. The registration map is applied to one of the first and second images to register the one image to the other and propagate objects of interest over.