A system and method for non-speech input or keypad-aided word and spelling recognition is disclosed. The method includes generating an unweighted grammar, selecting a database of words, generating a weighted grammar using the unweighted grammar and a statistical letter model trained on the database of words, receiving speech from a user after receiving the non-speech input and after generating the weighted grammar, and performing automatic speech recognition on the speech and non-speech input using the weighted grammar. If a confidence is below a predetermined level, then the method includes receiving non-speech input from the user, disambiguating possible spellings by generating a letter lattice based on a user input modality, and constraining the letter lattice and generating a new letter string of possible word spellings until a letter string is correctly recognized.