A system and method are provided for model-based coronary artery calcium (CAC) scoring. A model image of a heart region is aligned with an image of a patient's heart region in order to more easily identify the coronary arteries and other relevant anatomical features in the image. Once the images are aligned, relevant calcium plaques are identified by their presence within a coronary artery, and the relevant plaques are then labeled by the specific coronary artery in which they are located. The coronary arteries with the labeled plaques are scored individually based on their size and X-ray attenuation, and an overall score based on all of the relevant plaques is then computed, which is related to the patient's risk for coronary artery disease.