A system and method providing gesture controlled video conferencing includes a local capture device detecting movements of a user in a local environment and an audio/visual display. A processor is coupled to the capture device and a remote capture device and a remote processor at a remote environment via a network. The local processor includes instructions to render a representation of the remote environment on the display responsive to the remote processor and remote capture device. The processor also tracks movements of a local user in a space proximate to the local capture device. Responsive to a user gesture detected at the local capture device, the audio or visual signals provided by the remote capture device are altered to change the representation of the remote location is altered locally.