The Framework for Machine Translation Evaluation
in ISLE is a resource that helps MT evaluators define
contextual evaluation plans. FEMTI consists of two
interrelated classifications or taxonomies: the first one
lists possible characteristics of the contexts of use that
are applicable to MT systems. The second one lists the
possible characteristics of an MT system, along with the
metrics that were proposed to measure them.
Evaluators using FEMTI specify the intended context of use
for an MT system using the first classification, and submit it
to FEMTI. In return, FEMTI proposes a set of quality
characteristics that are relevant to that context, using its
embedded knowledge base. Evaluators can modify this set of
quality characteristics and select evaluation metrics for each
of them, by browsing the second classification. Evaluators can
then print the evaluation plan and execute the evaluation.
The main characteristics of the translation task are:
- Assimilation: The ultimate purpose of the assimilation task
(of which translation forms a part) is to monitor a (relatively)
large volume of texts produced by people outside the
organization, in (usually) several languages. - Dissemination: The ultimate purpose of dissemination is to
deliver to others a translation of documents produced inside
the organization. - Communication: The ultimate purpose of the communication
task is to support multi-turn dialogues between people who speak
different languages. The translation quality must be high enough
for painless conversation, despite possible syntactically ill-formed
input and idiosyncratic word and format usage.
