Mechanisms are provided, in a smart speaker system having at least one smart speaker device comprising an audio capture device, and smart speaker system logic, for processing audio sample data captured by the audio capture device. The audio capture device captures an audio sample from a monitored environment and one or more sensor devices capture sensor data representing non-verbal attention indicators associated with a speaker of a speech portion of the audio sample. The smart speaker system logic evaluates the non-verbal attention indicators of the sensor data to determine whether or not the speech portion of the audio sample is directed to the smart speaker device. In response to determining that the speech portion of the audio sample is directed to the smart speaker device, a cognitive system associated with the smart speaker system generates a response to the speech portion.