A portable apparatus for controlling a robot and a method therefor. The portable apparatus includes: an orientation sensor adapted for measuring orientation of the portable apparatus; an HMI device adapted for detecting two-dimensional manual motion relative to the HMI device; and a processing unit adapted for receiving a first signal representing the measured orientation of the portable apparatus and a second signal representing the detected two-dimensional manual motion relative to the HMI device and controlling a part of the robot to move in a direction in consideration of the measured orientation of the portable apparatus and the detected two-dimensional manual motion relative to the HMI device. By having the portable apparatus and the method therefor as explained herein, the two-dimensional manual movement on touch panel is integrated with orientation of the portable apparatus and an integration of these is mapped by the robot, which makes it possible to define a path in three dimensional space for jogging/teaching robot's movements in three dimension.