A method operating a Graphics Processing Unit (GPU) memory can be provided by accessing specified training parameters used to train a Deep Neural Network (DNN) using a GPU with a local GPU memory, the specified training parameters including at least a specified batch size of samples configured to train the DNN. A sub-batch size of the samples can be defined that is less than or equal to the specified batch size of samples in response to determining that an available size of the local GPU memory is insufficient to store all data associated with training the DNN using one batch of the samples. Instructions configured to train the DNN using the sub-batch size can be defined so that an accuracy of the DNN trained using the sub-batch size is about equal to an accuracy of the DNN trained using the specified batch size of the samples.