A method is provided for managing a machine learning system. In the method, a database is provided for storing a plurality of data elements. A plurality of machine learning models is trained using assigned subsets of the plurality of data elements. The outputs of the plurality of machine learning models is provided to an aggregator. During inference operation of the machine learning system, the aggregator determines a final output based on outputs from the plurality of models. If it is determined that an assigned subset must be changed because, for example, a record must be deleted, then the data element is removed from the selected assigned subset. The affected machine learning model associated with the changed assigned subset is removed, and retrained using the changed assigned subset.