An audio data processing device includes a mixing unit that mixes first audio data and second audio data to generate mixed audio data, a first output unit that outputs, to a first speaker, audio data not having undergone the mixing, a second output unit that outputs, to a second speaker, the mixed audio data generated by the mixing unit, a processor which is provided between a point of branching to the mixing unit and the first output unit or between the mixing unit and the second output unit and adds an acoustic effect, and a delay unit which is provided between the branching point and the first output unit or between the mixing unit and the second output unit and delays the audio data so that a relationship of timing between input, to the first speaker, of the audio data output from the first output unit and input, to the second speaker, of the audio data output from the second output unit coincides with a relationship of timing of input between the first audio data and the second audio data.