Systems and computer-implemented methods include a validation phase in which validation projects are submitted to an online crowdsourcing system. The validation projects each have different combinations of the two validation control parameters, payment per human intelligence task (“HIT”) and number of solutions per HIT. By analyzing the solutions for the validation projects, final control parameters can be selected from the validation project having the best combination of high quality, low cost, and short time to completion. The final control parameters can be submitted along with a final project to the online crowdsourcing system. In addition, a novel outlier detection algorithm enables the system to detect outlier workers based on the distribution of non-experts' agreement probabilities, reject outlier workers' results, and thus save labeling cost.