A method and system perform single phase and multi-phase contour refinement of lesions. The method includes receiving a three dimensional input mask; receiving input slices from the medical images including a lesion; cropping the input slices with the input mask; performing lesion contour refinement for the cropped input slices and the input mask to obtain a predicted mask; and storing the predicted mask that includes 3D lesion contour refinement. A multiphase method includes deforming the 3D input mask from the reference phase to a target phase or warping the input slices from the target phase to the reference phase before contour refinement. The warped images generate an output mask in the reference phase coordinate system that is then deformed to the target phase coordinate system for display.