Techniques are disclosed for generating audio based on visual information. In some examples, an audio generation system is trained using supervised learning using a training set generated from videos. The trained audio generation system is able to infer audio for provided silent video based on the visual contents of the silent video, and generate raw waveform samples that represent the inferred audio.