A system and method for speech recognition is provided. Embodiments may include receiving an audio signal at a first deep neural network (“DNN”) associated with a computing device. Embodiments may further include receiving the audio signal at a second deep neural network (“DNN”) associated with a computing device, wherein the second deep neural network includes fewer parameters than the first deep neural network. Embodiments may also include determining whether to select an output from the first deep neural network or the second deep neural network and providing the selected output to a decoder with an overall objective of speeding up ASR.