Gesture recognition system for remote controlling a TV is disclosed. The system comprises: (a) a webcam for capturing a video sequence of a user which is composed of video frames (b) a hand tracker for receiving the video sequence, and when a hand is determined within a frame of the video sequence, the hand tracker calculates the hand centroid; (c) a trajectory buffer for receiving plurality of the hand centroids as determined from plurality of frames respectively; (d) trajectory segmentation module for continuously inspecting the buffer for possible inclusion of relevant trajectories that relate to a hand gesture pattern, and whenever a relevant trajectory is detected, it extracts a respective trajectory segment, and conveys the segment into a gesture classifier; and (e) the gesture classifier for verifying whether the segment relates to a specific gesture command, and when affirmative, the gesture classifier transfers the respective command to the TV.