A method for building a robust classifier against evasion attacks includes receiving an application, identifying one or more features of the application, and determining a first confidence score for a first version of the application including a first set of features and determining a second confidence score for a second version of the application including a second set of features, the first set of features is different than the second set of features. The method also includes determining a difference between the first confidence score and the second confidence score, and comparing the difference with a convergence threshold. The method includes, based on the comparison, determining whether the first confidence score exceeds a confidence score threshold, and generating a report based on determining the first confidence score exceeds the confidence score threshold.