A method and system for extracting motion-based layers from fluoroscopic image sequences are disclosed. Portions of multiple objects, such as anatomical structures, are detected in the fluoroscopic images. Motion of the objects is estimated between the images is the sequence of fluoroscopic images. The images in the fluoroscopic image sequence are then divided into layers based on the estimated motion. In a particular implementation, the coronary vessel tree and the diaphragm can be extracted in separate motion layers from coronary angiograph fluoroscopic image sequence.