Provided is a method of authenticating a user by correlating speech and corresponding lip shape. An audiovisual of a user requesting authentication is captured. The audiovisual is processed to generate a speech vector quantization sequence and a corresponding lip vector quantization sequence of the user. A likelihood of the speech vector quantization sequence and the corresponding lip vector quantization sequence with probability distributions of speech vector quantization code words corresponding to different lip shape vector quantization code words of the user requesting authentication weighed by probabilities of speech and lip vector quantization indices of the user requesting authentication is evaluated. If upon evaluation, a likelihood of the user requesting authentication being an authentic user is more than a predefined threshold, the user is authenticated.