A speech signal processing apparatus includes a control signal output unit configured to receive as an input signal either one of a first speech signal corresponding to a sound uttered by a user and a second speech signal corresponding to a sound output from an eardrum of the user when the user utters a sound, and output a control signal corresponding to a noise level of the input signal, and a speech signal output unit configured to output either one of the first speech signal and the second speech signal according to the control signal.