A first matrix (W(k)) indicating frequency characteristics of a separation filter is calculated from input signals of a plurality of channels. A second matrix (Ws(k)) is calculated by using the restriction coefficients (Ci(k)) for restricting the separation filter and the first matrix, and separation filter coefficients (wsij(s)) are calculated by using the second matrix. With use of the separation filter coefficients, separation signals (ysi(t)) are then calculated from the input signals. A third matrix (Ws−1(k)) is then calculated by transforming the second matrix into an inverse matrix at each frequency, and reproduction filter coefficients (a′I1(s), a′I2(s)) are calculated by using the third matrix. With use of the reproduction filter coefficients, the synthesized signal of each channel is calculated by using the separation signals. The restriction coefficients are calculated so that the reproduction filter coefficients indicate filter coefficients which perform a sound source localization on the separation signals.