Embodiments of the disclosure provide systems and methods for detecting a medical condition of a subject. The system includes a communication interface configured to receive a sequence of images acquired from the subject by an image acquisition device and an end-to-end multi-task learning model. The end-to-end multi-task learning model includes an encoder, a Convolutional Recurrent Neural Network (ConvRNN), and at least one of a decoder and a classifier. The system further includes at least one processor configured to extract feature maps from the images using the encoder, capture contextual information between adjacent images in the sequence using the ConvRNN, and detect medical condition of the subject using the classifier based on the extracted feature maps of the image slices and the contextual information or segment each image slice using the decoder to obtain a region of interest indicative of the medical condition based on the extracted feature maps.