A voice conversion device includes: a parameter learning unit in which a probabilistic model that uses speech information, speaker information, and phonological information as variables to thereby express relationships among binding energies between any two of the speech information, the speaker information and the phonological information by parameters is prepared, wherein the speech information is obtained based on a speech, the speaker information corresponds to the speech information, and the phonological information expresses the phoneme of the speech, and in which the parameters are determined by performing learning by sequentially inputting the speech information and the speaker information into the probabilistic model; and a voice conversion processing unit that performs voice conversion processing of the speech information obtained on the basis of the speech of an input speaker, based both on the parameters determined by the parameter learning unit and on the speaker information of a target speaker.