A system, method, and apparatus for a robot system that manipulates the surface of an object effect programmed manipulation goals such as reaching specific locations on the surface of the object, displacing the surface of the object, applying a predetermined force and torque to the surface of the object, dynamically changing the contact point between the robot and the object, and applying force to structures below the surface of the object. The system and method determine the state of the object through a sensing method that includes, without limitation: torque and force measurement, visible light sensors, range and depth sensors, ultrasound sensors, thermographic sensors, and worktable force measurement.