A recommender system determines a probability threshold for an activity-prediction model, and uses the probability threshold to predict whether a user is performing a target activity. To determine the probability threshold, the system computes a set of activity probabilities based on contextual information for a set of historical activities, and based on an activity-prediction model for a target activity. The system then compares a set of probability thresholds with the set of activity probabilities to determine a prediction success rate for each probability threshold. The system computes a utility score for each probability threshold based on the prediction success rates and a utility function, and selects a probability threshold whose utility score is optimal amongst the utility scores of the set of thresholds and greater than or equal to a baseline utility score. The system then assigns the probability threshold to the activity-prediction model.