Deep machine learning methods and apparatus related to the manipulation of an object by an end effector of a robot are described herein. Some implementations relate to training an action prediction network to predict a probability density which can include candidate actions of successful grasps by the end effector given an input image. Some implementations are directed to utilization of an action prediction network to visually servo a grasping end effector of a robot to achieve a successful grasp of an object by the grasping end effector.