Some embodiments relate to using an unnormalized neural network language model in connection with a speech processing application. The techniques include obtaining a language segment sequence comprising one or more language segments in a vocabulary; accessing an unnormalized neural network language model having a normalizer node and an output layer comprising a plurality of output nodes, each of the plurality of output nodes associated with a respective language segment in the vocabulary; and determining, using the unnormalized neural network language model, a first likelihood that a first language segment in the vocabulary follows the language segment sequence.