The present disclosure relates to techniques for segmenting objects within medical images using a deep learning network that is localized with object detection based on a derived contrast mechanism. Particularly, aspects are directed to localizing an object of interest within a first medical image having a first characteristic, projecting a bounding box or segmentation mask of the object of interest onto a second medical image having a second characteristic to define a portion of the second medical image, and inputting the portion of the second medical image into a deep learning model that is constructed as a detector using a weighted loss function capable of segmenting the portion of the second medical image and generating a segmentation boundary around the object of interest. The segmentation boundary may be used to calculate a volume of the object of interest for determining a diagnosis and/or a prognosis of a subject.