Parallel training of a machine learning model on a computerized system is described. Computing tasks of a system can be assigned to multiple workers of the system. Training data can be accessed. The machine learning model is trained, whereby the training data accessed are dynamically partitioned across the workers of the system by shuffling subsets of the training data through the workers. As a result, different subsets of the training data are used by the workers over time as training proceeds. Related computerized systems and computer program products are also provided.