A system and method for offloading augmented reality processing is described. A sensor external to a viewing device of a user tracks a location and an orientation of the viewing device. The location and orientation are defined relative to predefined references of a physical environment local to the user. A server receives a request from the viewing device to offload at least one of a tracking process and an augmented reality rendering process. The augmented reality rendering process is based on an augmented reality database. The server generates offloaded processed data based on the request and the location and the orientation of the viewing device. The offloaded processed data is streamed to the viewing device. A visualization of the offloaded processed data is generated in the viewing device.