An apparatus for recognizing a user's gesture for carrying out an operation of a vehicle may include: an image generator that generates an object image by photographing a user's hand; a feature point candidate detector that generates a hand image including a hand region by eliminating a background of the object image; a palm detector that detects a palm region by performing a morphology operation on the hand region; a finger detector that detects a finger region by eliminating the palm region from the hand region; a feature point detector that detects the finger valleys located within the palm region and the fingertips located within the finger region as feature points; and a gesture recognizer that recognizes a number of unfolded fingers based on the hand model, and recognizes the user's gesture by selecting a category that corresponds to the number of unfolded fingers among a plurality of categories.