Embodiments herein describe an audio system that adjusts based on the location of a person. That is, instead of relying on fixed speakers, the audio system adjusts the direction of audio output for one or more speakers to optimize the performance of the audio system based on the location of a user or based on the number of users. To do so, the audio system may include a camera and a tracking application which identifies the location of a user and/or the number of users in front of the camera. Using this information, the audio system adjusts one or more actuators coupled to a speaker to change the direction of the audio output of the speaker. As the user continues to move or shift, the audio system can continually adjust the speaker to optimize the performance of the system.