The disclosed embodiments provide a system that facilitates selecting a message to be presented to users based on a statistically valid hypothesis test. During operation, the system runs a hypothesis test by presenting alternate versions of a message to a test set of users and receives user-feedback data. Next, the system obtains a significance level for the test and determines a number of independent data subsets associated with data from the test. The system subsequently uses the significance level and the number of independent data subsets to calculate an individual significance level for each independent data subset. The system then uses the individual significance levels to calculate an amount of user-feedback data required to achieve the significance level during the test, and selects one of the alternate versions of the message by analyzing the calculated amount of user-feedback data during the test. Finally, the system presents the selected version.