Systems and methods for classifying abnormalities within optical coherence tomography images of the eye are presented. One embodiment of the present invention is the classification of pigment epithelial detachments (PEDs) based on characteristics of their internal reflectivity, size and shape. The classification can be based on selected subsets of the data located within or surrounding the abnormalities. Training data can be used to generate the classification scheme and the classification can be weighted to highlight specific classes of particular clinical interest.