A scalable rank filter and method for performing rank filtering are disclosed. In one embodiment, the rank filter comprises a W staged pipeline with W stages to receive N input data samples and operable to generate an output based on the N input data samples as a result of the W stages completing execution, where W is a bit length of the inputs and W and N are integers greater than two; and output logic coupled to the W staged pipeline to determine the output prior to all W stages completing execution and to output the median.