Multiple person viewing of rendered medical image is achieved by manipulating an image by the server to produce first rendered medical image data that includes the medical image data plus image manipulation results. A first 3-D medical image that is based on the first rendered medical image data is displayed within an augmented reality environment or a virtual reality environment so that the first end user has a first viewer location and a first viewing angle of the first 3-D medical image. A second 3-D medical image that is based on second rendered medical image data is displayed within the augmented reality environment or the virtual reality environment so that the second end user has at least one of a second viewer location and a second viewing angle that is different from at least one of the first viewer location and the first viewing angle of the first 3-D medical image.