How to evaluate the quality of MT output

Overview

This half-day workshop provides a broad overview of how to evaluate translations that are produced by machine translation systems. The range of issues covered includes a broad survey of both human evaluation measures and commonly-used automated metrics, and a review of how these are used for various types of evaluation tasks, such as assessing the translation quality of MT-translated sentences, comparing the performance of alternative MT systems, or measuring the productivity gains of incorporating MT into translation workflows.

The workshop is suitable and important for anyone planning to or already using MT within their translation operations.

Cost

Non-members: 450 euros

Members: 300 euros

All participants receive a comprehensive set of notes and a complimentary copy of the TAUS Report "How to evaluate the quality of MT output".

Outline

• Introduction: the importance of MT evaluation
• Human MT Evaluation Measures
• Automated MT Evaluation Metrics
• MT Evaluation within Commercial Translation Operations

Audience

Anyone planning to or already using MT within their commercial translation operations. In particular, IT and commercial translation operation and project managers that are considering using MT.

Level

Intermediate - participants should be familiar with localization processes and be minimally familiar with how MT systems operate

Following the workshop participants will

• Understand how and in what ways translation quality of MT systems can be assessed
• Understand how to conduct human assessments of MT translation quality, which human measures to use, and how to properly analyze results of such evaluations
• Have a clear conceptual understanding of the major automated metrics available (BLEU, METEOR, TER), how and when to use them, and which metric is most appropriate for various evaluation tasks.
• Be able to effectively compare different MT engines on a representative set of documents
• Be able to effectively contrast MT engine translation performance before and after MT system development and/or customization
• Understand the limitations of current MT evaluation measures and metrics and tailor the available technology to their individual needs

Workshop Leader

Dr. Alon Lavie, Associate Research Professor at Carnegie Mellon University, AMTA President, and President of Safaba Translation Solutions.