A clothing search system provides a clothing search to users using a component-based image search. Retailer catalogs are analyzed to determine clothing components within clothing images. Features associated with the components are determined. When a user requests a clothing search, the clothing search system selects clothing based on the components and features requested by the user. The user may also provide an image to the clothing search system. The clothing search system determines components and features of the image and identifies clothing with matching components.