A system for registering and segmenting images includes an image scanner configured to acquire an image pair including a first image at a first time and a second image at a second time that is after the first time. A joint registration and segmentation server receives the image pair from the image scanner and simultaneously performs joint registration and segmentation on the image pair using a single deep learning framework. A computer vision processor receives an output of the joint registration and segmentation server and characterizes how a condition has progressed from the first time to the second time therefrom. A user terminal presents the characterization to a user.