A method for recognizing heart diseases in a cardiac echo video of a heart with an unknown disease using a spatio-temporal disease model derived from a training echo video, comprising the steps of: generating a plurality of training models for heart diseases, wherein the cardiac echo videos are each derived from a known viewpoint and the disease of the heart is known; analyzing the video of the heart with the unknown disease by fitting a model of shape and motion for each frame and combining the results across the frames; and, reporting the disease using a classification method for choosing among the diseases of interest.