A medical image viewing system allows multiple person viewing of rendered medical images, the medical image viewing system. A medical image processing system receives medical image data and manipulates the medical image data to produce first rendered medical image data that includes the medical image data plus image manipulation results. A first client system receives the first rendered medical image data from the medical image processing system. The first client system displays to a first end user on the first client system, a first 3D medical image that is based on the first rendered medical image data. The first 3D medical image is displayed within an augmented reality environment or a virtual reality environment so that within the augmented reality environment or the virtual reality environment the first end user has a first viewer location and a first viewing angle of the first 3D medical image.