A method for medical image segmentation. The method includes accessing and updating a knowledge-base in accordance with embodiments of the present invention. The techniques include: receiving a medical image and computing a sparse landmark signature based on the medical image content. Next, a knowledge-base is searched for representative matches to form a base set, wherein the base set comprises a plurality of reference image sets. A portion of the plurality of reference image sets of the base set is deformed to generate mappings from the base set to the medical image set. Finally a weighted average segmentation for each structure of interest of the medical image set is determined.