A body-worn hearing assist device such as a hearing aid allows at least two audio inputs, such as through a wireless chipset and from an onboard microphone. The hearing aid has a directional sensor, preferably a combination magnetic sensor/accelerometer, which allows the hearing aid to determine which direction the user is facing. A directional reference associated with a “remote” sound source, i.e., the direction the user would naturally face to best hear the wirelessly transmitted audio, is stored in memory. When the user faces in the direction of the “remote” sound source, such as watching a screen of an audio-visual program having a wireless audio streamer or looking at someone using a wireless microphone, the wirelessly received audio is coupled as the primary input of the hearing aid. When the user turns and faces a different direction, the hearing aid automatically switches the primary input source to the onboard microphone.