Systems and methods for generating synthesized medical images for training a machine learning based network. An input medical image in a first modality is received comprising a nodule region for each of one or more nodules, a remaining region and an annotation for each of the nodules. A synthesized medical image in a second modality is generated from the input medical image comprising the annotation for each of the nodules. A synthesized nodule image of each of the nodule regions and synthesized remaining image of the remaining region are generated in the second modality. It is determined whether a particular nodule is visible in the synthesized medical image based on the synthesized nodule image for the particular nodule and the synthesized remaining image. If at least one nodule is not visible in the synthesized medical image, the annotation for the not visible nodule is removed from the synthesized nodule image.