A stain separation system of digital pathology images that performs transforming a digital image from a first color domain to an optical domain to form an optical domain image (ODI), identifying a plane containing two or more basis vector which contain the pixels of the ODI, determining a plurality of orthogonal vector within the identified plane, forming a histogram of the digital image represented by the orthogonal vectors and determining one or more final stain vectors by searching for candidate vectors in the plane that minimize a cost function of the histogram.