A method of tracking an object performed in a portable terminal linked with a rotatable cradle so that a rotation of the rotatable cradle is controlled to track the object, and the method includes receiving frame images captured in a forward direction at a preset frame rate, and detecting whether an object exists in the frame image for each of the frame images, when the object is detected in the frame image for each of the frame images, calculating a reference distance in association with a position of the detected object in the frame image, calculating a rotational speed of the rotatable cradle for a first frame image based on accumulation of reference distances of a plurality of frame images included in a frame group including the first frame image, and controlling driving of the rotatable cradle based on the rotational speed.