In many medical image classification problems, distinctive image features are often localized in certain anatomical regions. The key to efficient and accurate classification in such problems is the localization of the region of interest (ROI). To address this problem, a multi-atlas label fusion technique was developed for automatic ROI detection. Given training images with class labels, the present method infers voxel-wise scores for each image showing how distinctive each voxel is for categorizing the image. The present method for ROI segmentation and for class specific ROI patch extraction in a 2D cardiac CT body part classification application was applied and shows the effectiveness of the detected ROIs.