One embodiment of the present invention provides a system for recognizing and classifying clothes. During operation, the system captures at least one image of a clothing item. The system further determines a region on the captured image which corresponds to a torso and/or limbs. The system also determines at least one color composition, texture composition, collar configuration, and sleeve configuration of the clothing item. Additionally, the system classifies the clothing item into at least one category based on the determined color composition, texture composition, collar configuration, and sleeve configuration. The system then produces a result which indicates the classification.