There is provided a signal processing device that includes a relative azimuth prediction unit that predicts, on the basis of a delay time in accordance with a distance from a virtual sound source to a listener, a relative azimuth of the virtual sound source when a sound of the virtual sound source reaches the listener, and a binaural-room impulse response (BRIR) generation unit that acquires a head-related transfer function of the relative azimuth for each one of a plurality of the virtual sound sources and generates a BRIR on the basis of a plurality of the acquired head-related transfer functions.