Inter-operative switching of tools in a robotic system includes a system with a plurality of manipulators and a controller. The controller is configured to detect mounting of a first imaging device to a first manipulator of the plurality of manipulators, the first imaging device having a first reference frame; in response to detecting the mounting of the first imaging device, control a tool relative to the first reference frame using a second manipulator of the plurality of manipulators, the tool being mounted to the second manipulator; detect mounting of a second imaging device to a third manipulator of the plurality of manipulators, the second imaging device having a second reference frame; and in response to detecting the mounting of the second imaging device, control the tool relative to the second reference frame using the second manipulator.