Aspects of the disclosure provide a method for crowd segmentation that can globally optimize crowd segmentation of an input image based on local information of the input image. The method can include receiving an input image of a site, initializing a plurality of hypothesis based on the input image, dividing the input image into a plurality of patches, calculating an affinity measure of one or more patches to a hypothesis based on a partial response of the patches to a whole body classifier of the hypothesis that includes a combination of weak classifiers, and optimizing assignments of the plurality of patches to the plurality of hypothesis based on the affinity measures of the plurality of patches to the plurality of hypothesis.