A mechanism is disclosed for evaluating the quality of a machine translation system. Propositions are identified within a source language string and within a translation of the source language string generated by a machine translation system. The propositions can be identified using an automated mechanism or using human translators through the use of a human intelligence task site. Once the propositions have been identified, the propositions identified in the source language string can be compared to the propositions identified in the translated target language string. The results of the comparison can be utilized to compute a quality score for the translation. A final quality score can be generated for the machine translation system by repeating this process for multiple source language strings. The final quality score can then be utilized to improve the quality of models utilized by the machine translation system.