Systems and methods for assessing expansion of an abnormality are provided. A first input medical image of a patient depicting an abnormality at a first time and a second input medical image of the patient depicting the abnormality at a second time are received. The second input medical image is registered with the first input medical image. The abnormality is segmented from 1) the first input medical image to generate a first segmentation map and 2) the registered second input medical image to generate a second segmentation map. The first segmentation map and the second segmentation map are combined to generate a combined map. Features are extracted from the first input medical image and the registered second input medical image are based on the combined map. Expansion of the abnormality is assessed based on the extracted features using a trained machine learning based network. Results of the assessment are output.