A method of generating a category model for classifying medical images. The method comprises providing a plurality of medical images each categorized as one of a plurality of categorized groups, generating an index of a plurality of visual words according to a distribution of a plurality of local descriptors in each the image, modeling a category model mapping a relation between each visual word and at least one of the categorized groups according to the index, and outputting the category model for facilitating the categorization of an image based on local descriptors thereof.