A computer-based system and method to improve the multimodal fusion output at the decision level is disclosed. The method proposes computation of a confidence weighted measure for the individual score values obtained for each modality and fuse these new updated scores to get the final decision. These confidence weights are the performance parameters (measured in terms of F-measure) during the offline training step. The process significantly increases the accuracy of the multimodal system.