A voice dialog device, comprises a sight line detection unit configured to detect a sight line of a user; a voice processing unit configured to obtain voice pronounced by the user and a result of recognizing the voice; a dialog determination unit configured to determine whether or not the voice dialog device has a dialog with the user; and an answer generation unit configured to generate an answer, based on a result of recognizing the voice, wherein the dialog determination unit determines whether or not the user has started the dialog, based on both the sight line of the user and the obtained voice.