A method of operating a hearing assistance system that includes a binaural hearing device with first and second earpieces, each earpiece adapted to be worn over, on or in one ear. Each earpiece has an electroacoustic transducer that converts an input electrical audio signal for the ear into sound, and at least one microphone that converts sound into an electrical microphone output signal. The hearing assistance system is adapted to process microphone output signals from both earpieces to create the electrical audio signals that are used to drive the transducers of both earpieces. A situation in which one of the earpieces is not over, on or in an ear is detected, and in response the processing of at least one of the microphone output signals that creates the audio signals is modified.