Systems and methods are disclosed for performing operations comprising: receiving first and second images depicting an anatomy of a subject; obtaining a segmentation associated with the first image; applying a trained neural network to estimate the adapted segmentation corresponding to the anatomy depicted in the second image, the trained network consisting of three sub-networks: a registration sub-network, generating an initial segmentation estimate representing a deformation of the segmentation associated with the first image to fit the anatomy depicted in the second image, a segmentation sub-network, generating a second initial segmentation estimate for the second image, and a third refinement sub-network, combining the two initial segmentations and generating a refined segmentation for the second image.