Systems and methods are provided for performing speech to intent classification. The systems and methods perform operations comprising: receiving an audio file comprising speech input; processing, by a speech recognition engine, the audio file comprising the speech input to generate an initial character-based representation of the speech input; processing, by an intent classifier, the initial character-based representation of the speech input to generate an estimated intent of the speech input; and generating, by the speech recognition engine, a textual representation of the speech input based on the estimated intent of the speech input.