A signal processing device is provided, including a prediction error calculating unit that calculates an error signal between a left signal l(n) and a prediction signal of the left signal l (n) predicted from a right signal r(n), a gain adjusting unit that makes a gain adjustment and outputs an error signal, a first adder that adds the left signal l(n) and the error signal and outputs, and a second adder that adds the right signal r(n) and the error signal in opposite phase and outputs.