Disclosed herein are system, method, and computer program product embodiments for switching between local and remote guidance instructions for autonomous vehicles. For example, the method includes, in response to monitoring one or more actions of objects detected in a scene in which the autonomous robotic system is moving, causing the autonomous robotic system to slow or cease movement in the scene. The method includes detecting a trigger condition based on movement of the autonomous robotic system in the scene. In response to the one or more monitored actions and detecting the trigger condition, the method includes transmitting a remote guidance request to a remote server. After transmitting the remote guidance request, the method includes receiving remote guidance instructions from the remote server and causing the autonomous robotic system to begin operating according to the remote guidance instructions.