Contact center agents often work in close proximity to other agents. As a primary agent is engaged in a call, a neighboring agent speech may be picked up by the primary agent's microphone. Contact centers using automated speech recognition systems may monitor the agent's speech for key terms and, if detected, respond accordingly. Determining a primary agent spoke a key term, when the true speaker of the key term is a neighboring agent, may cause errors or other problems. Characterizing at least the primary agent's voice and then, once a key term is detected, determining if it was the primary agent that spoke the key term, may help to reduce the errors. Additionally, computational requirements may be reduced as non-key terms may be quickly discarded and optionally, key terms determined to not have been spoke by the primary agent, may also be discarded without further processing.