An image based motion gesture recognition method and system thereof are disclosed. In embodiment, a hand posture detection is performed according to the received image frames, to obtain a first hand posture. It is then determined whether the first hand posture matches a predefined starting posture or not. If the first hand posture matches said predefined starting posture, movement tracking is performed according to hand locations on image frames, to obtain a motion gesture. During said movement tracking, the hand posture detection is performed according to said image frames to obtain a second hand posture, and it is determined whether the second hand posture matches a predefined ending posture. If the second hand posture matches the predefined ending posture, the movement tracking is stopped. Therefore, reduce complexity motion gesture recognition can be reduced and the reliability in interaction can be improved.