Various implementations include systems for processing audio signals to remove artifacts introduced by a machine learning system in challenging environments. In particular implementations, a method includes generating a processed audio signal for a hearing assistance device in which the processed audio signal is intended to perceptually dominate a user auditory experience, including: processing an unprocessed audio signal received by the hearing assistance device, wherein the processing includes utilizing a machine learning (ML) system to generate an ML enhanced audio signal; determining a mixing coefficient from an environmental noise assessment; mixing the ML enhanced audio signal with the unprocessed audio signal using the mixing coefficient to generate the processed audio signal; and outputting the processed audio signal.