An artificial neural network (ANN) generates a base expanded matrix that represents an output of a layer of the ANN, such as the output layer. Values in each row are grouped with respect to a set of network parameters in a previous layer, and a sum of the values in each row produces an output vector of activations. The ANN updates the values in at least one column of the expanded matrix according to parameter updates, which results in an updated expanded matrix or an update expanded matrix. An error or a total cost can be computed from the updated expanded matrix or the update expanded matrix. Nonlinear activation functions can be modeled as piecewise linear functions, and a change in an activation function's slope can be modeled as a linear update to an expanded matrix. Parameter updates can be constrained to a restricted value set in order to simplify update operations performed on the expanded matrices.