A robotic surgery method includes tracking a position of a surgical tool as it is manually manipulated to perform a procedure. The tool is coupled to a handheld manipulator assembly, and the handheld manipulator assembly includes a handheld portion configured to be manually supported and moved by a user and a tool drive assembly supported by the handheld portion. The manipulator assembly further includes a plurality of elongate structural members coupled between the tool drive assembly and the handheld portion, at least one pivotal link arranged between the tool drive assembly and the plurality of elongate structural members, a plurality of lead screws and actuators supported by the handheld portion, and a controller in communication with the plurality of actuators. The method further includes selectively operating the actuators to move the tool drive assembly relative to the handheld portion based on the tracked position of the tool.