Systems and methods for performing non-obtrusive, automatic adjustment of an auditory prosthesis are disclosed. A control expression can be detected during a conversation. Upon detection of the control expression, an audio setting adjustment can be selected and applied to the auditory prosthesis. Multiple adjustments can be made in response to identifying multiple control expressions during a conversation.