The invention discloses a control apparatus capable of providing a user with a control over an object displayed on a display apparatus by a pointing device. The control apparatus includes an image capturing module, a separating module, a positioning module, a constructing module, and a processing module. The image capturing module is used to record an image sequence included N images. The separating module is applied to capture the pointing device image related to the pointing device from each image. The positioning module is used for calculating a specific point of the pointing device image of each image to generate a first set of specific point information. The constructing module is applied to generate a trajectory in accordance with a pre-defined criterion and the first set of specific point information. Additionally, the processing module is used to analyze the trajectory and generate a control signal to control the object.