In one embodiment, images of a first user in a video conference are captured with one or more physical video cameras. The captured images are processed to form a three-dimensional (3-D) model of the first user. A location on a display screen is determined where an image of each of one or more second users in the video conference is shown. One or more virtual cameras are positioned in 3-D space. Each virtual camera is associated with a respective second user and positioned in 3-D space based on the location on the display screen where the image of the associated second user is shown. A view of the first user from the perspective of each of the one or more virtual cameras is rendered. The rendered view of the first user from the perspective of each virtual camera is shared with the associated second user for the respective virtual camera.