Joint sound model generation techniques are described. In one or more implementations, a plurality of models of sound data received from a plurality of different sound scenes are jointly generated. The joint generating includes learning information as part of generating a first said model of sound data from a first one of the sound scenes and sharing the learned information for use in generating a second one of the models of sound data from a second one of the sound scenes.