A method and apparatus for vascular disease detection and characterization using a recurrent neural network (RNN) is disclosed. A plurality of 2D cross-section image patches are extracted from a 3D computed tomography angiography (CTA) image, each extracted at a respective sampling point along a vessel centerline of a vessel of interest in the 3D CTA image. Vascular abnormalities in the vessel of interest are detected and characterized by classifying each of the sampling points along the vessel centerline based on the plurality of 2D cross-section image patches using a trained RNN.