Mechanisms are provided to implement a hybrid deep learning network. The hybrid deep learning network receives, from a imaging system, first input data specifying a non-annotated image. The hybrid deep learning network pre-processes the non-annotated image to generate second input data specifying a hint image and corresponding annotation data specifying salient regions of the hint image. The hybrid deep learning network processes the first input data and second input data to perform training of the hybrid deep learning network by targeting feature detection in the non-annotated image in the salient regions identified in the hint image. The trained hybrid deep learning network is used to process third input data specifying a new non-annotated image to thereby identify an object or structure in the new non-annotated image.