A tumor is tracked in sequences of bi-plane images by generating a set of segmentation hypotheses using a 3D model of the tumor, a bi-plane geometry, and a previous location of the tumor as determined from the pairs of biplane images. Volume prior probabilities are constructed based on the set of hypotheses. Seed pixels are selected using the volume prior probabilities, and a bi-plane dual image graph is constructed using intensity gradients and the seed pixels to obtaining segmentation masks corresponding to tumor boundaries using the image intensities to determine a current location of the tumor.