Systems, methods, and devices are disclosed for detecting an active speaker in a two-way conference. Real time audio in one or more sub band domains are analyzed according to an echo canceller model. Based on the analyzed real time audio, one or more audio metrics are determined from output from an acoustic echo cancellation linear filter. The one or more audio metrics are weighted based on a priority, and a speaker status is determined based on the weighted one or more audio metrics being analyzed according to an active speaker detection model. For an active speaker status, one or more residual echo or noise is removed from the real time audio based on the one or more audio metrics.