A signal processing device includes a first panel displacement control unit to which a first audio signal is input, a second panel displacement control unit to which a second audio signal is input, and a control unit configured to control the first panel displacement control unit and the second panel displacement control unit. The first panel displacement control unit includes a first gain adjustment unit configured to adjust a level of the first audio signal, the second panel displacement control unit includes a second gain adjustment unit configured to adjust a level of the second audio signal, and the control unit includes a correlation determination unit configured to determine presence or absence of a correlation between the first audio signal and the second audio signal, and a gain control unit configured to control a level adjustment amount in each of the first gain adjustment unit and the second gain adjustment unit on the basis of a determination result of the correlation determination unit.