A method and system for model based fusion pre-operative image data, such as computed tomography (CT), and intra-operative C-arm CT is disclosed. A first pericardium model is segmented in the pre-operative image data and a second pericardium model is segmented in a C-arm CT volume. A deformation field is estimated between the first pericardium model and the second pericardium model. A model of a target cardiac structure, such as a heart chamber model or an aorta model, extracted from the pre-operative image data is fused with the C-arm CT volume based on the estimated deformation field between the first pericardium model and the second pericardium model. An intelligent weighted average may be used improve the model based fusion results using models of the target cardiac structure extracted from pre-operative image data of patients other than a current patient.