System and method for using information extracted from intuitive multimodal interactions in the context of media for media tagging are disclosed. In one embodiment, multimodal information related to media is captured during multimodal interactions of a plurality of users. The multimodal information includes speech information and gesture information. Further, the multimodal information is analyzed to identify speech portions of interest. Furthermore, relevant tags for tagging the media are extracted from the speech portions of interest.