A signal processing apparatus that processes a plurality of audio signals obtained by acquiring a sound in a target area by performing sound acquisition by a plurality of sound acquisition units, comprising: a specification unit configured to specify a position of a sound source in the target area and positions and directivities of the plurality of sound acquisition units; and a selection unit configured to select, among the plurality of audio signals based on the sound acquisition by the plurality of sound acquisition units, an audio signal to be played back based on a degree of misalignment of the directivity of each of the plurality of sound acquisition units with respect to the specified position of the sound source.