Systems and methods for associating audio signals in an environment surrounding a voice-controlled system include receiving by a voice-controlled system through a microphone, an audio signal from a user of a plurality of users within an environment surrounding the microphone. The voice-controlled system determines a source location of the audio signal. The voice-controlled system determines a first user location of a first user and a second user location of a second user. The voice-controlled system then determines that the first user location correlates with the source location such that the source location and the first user location are within a predetermined distance of each other. In response, the voice-controlled system performs at least one security action associated with the first user providing the audio signal.