A system and method of training an artificial neural network. The method comprises determining an activation value for each node in a set of nodes of the artificial neural network, the activation values being determined by applying training data to the artificial neural network, and scaling the determined activation values for each of a plurality of the nodes in a portion of the artificial neural network. Each scaled activation value is determined using a scaling factor associated with a corresponding one of the plurality of nodes. Each scaling factor is determined based on a rank of the corresponding node. The method further comprises updating weights associated with each of the plurality of nodes in the portion of the artificial neural network using the determined scaled activation values to train the neural network.