A surround sound outputting device includes a receiving portion which receives signals on a plurality of channels, a storing portion which stores measuring sound data representing a sound, an outputting portion which outputs a sound produced based on the signals on the plurality of channels or the measuring sound data in a controlled direction and in a beam shape, a controlling portion which controls a direction of the sound output from the outputting portion, a sound collecting portion which picks up the sound output from the outputting portion to produce picked-up sound data representing the picked-up sound, an impulse response specifying portion which specifies impulse responses in respective directions from respective sound data, a path characteristic specifying portion which specifies path distances of the paths through which the sounds output in the respective directions arrive at the sound collecting portion from the outputting portion and levels of the impulse responses based on the impulse responses in the respective directions, and an allocating portion which specifies directions satisfying a predetermined relationship between the path distances of the paths in the respective directions and the levels of the impulse responses with respect to the plurality of channels respectively, and allocates the signals on the plurality of channels to the specified directions. The controlling portion controls the outputting portion so that respective sounds based on the signals on the plurality of channels are output in the directions specified by the allocating portion.