Systems and methods are provided for accurately setting notification priority levels for applications on a user device. An example system includes a user device and a management server. When an application generates a notification, it provides a priority level for the notification. A management agent executing on the user device can detect the notification and its assigned priority level, determine a predicted priority level using a prediction engine or prediction server, and cause the application the replace or update the assigned priority level based on the predicted priority level. The management agent can then receive user actions related to that notification from the application, and use that information to determine an observed priority level. The prediction engine or prediction server can be updated based on the observed priority level, thereby dynamically increasing the accuracy of predictions.