A method of authenticating a user based on voice recognition of a keyword includes generating, at a processor, clean speech statistics. The clean speech statistics are generated from an audio recording of the keyword spoken by the user during an enrollment phase. The method further includes separating speech data and noise data from noisy input speech using the clean speech statistics during an authentication phase. The method also includes authenticating the user by comparing the speech data to the clean speech statistics or by comparing the noisy input speech to noisy speech statistics. The noisy speech statistics are based at least in part on the noise data.