A two-way conversation assisting device includes a first microphone that enters a first voice, a first loudspeaker that outputs the first voice, a second microphone that enters a second voice, a second loudspeaker that outputs the second voice, and a first echo and crosstalk canceller. The first echo and crosstalk canceller estimates and calculates, using an input signal into the second loudspeaker, a first interference signal indicative of degrees of a first echo caused when the second voice output from the second loudspeaker enters into the first microphone and first crosstalk caused when the second voice enters into the first microphone, and removes the calculated first interference signal from an output signal of the first microphone.