Described is a robotic control device for manipulating a gripper-held tool. The device includes a robotic gripper having a plurality of tactile sensors. Each sensor generates tactile sensory data upon grasping an tool based on the interface between the tool and the corresponding tactile sensor. In operation, the device causes the gripper to grasp a tool and move the tool into contact with a surface. A control command is used to cause the gripper to perform a pseudo-random movement with the tool against the surface to generate tactile sensory data. A dimensionality reduction is performed on the tactile sensory data to generate a low-dimensional representation of the tactile sensory data, which is then associated with the control command to generate a sensory-motor mapping. A series of control commands can then be generated in a closed-loop based on the sensory-motor mapping to manipulate the tool against the surface.