The subject disclosure is directed towards a noise adaptive beamformer that dynamically selects between microphone array channels, based upon noise energy floor levels that are measured when no actual signal (e.g., no speech) is present. When speech (or a similar desired signal) is detected, the beamformer selects which microphone signal to use in signal processing, e.g., corresponding to the lowest noise channel. Multiple channels may be selected, with their signals combined. The beamformer transitions back to the noise measurement phase when the actual signal is no longer detected, so that the beamformer dynamically adapts as noise levels change, including on a per-microphone basis, to account for microphone hardware differences, changing noise sources, and individual microphone deterioration.