A Long Road Ahead

Life vestQuality is the life vest of the translation industry: all players rely on the magical mystery word that instantly explain everything and forbid further questioning, as passengers on a plane feel confident knowing there is a life vest under their seat even though they could hardly use it in the case of an accident on pre-flight instructions.

Last Thursday, I attended the QTLaunchPad Workshop on Multidimensional Quality Metrics held in conjunction with the Sixth MultilingualWeb Workshop.

QTLaunchPad is a European Commission-funded collaborative research initiative dedicated to overcoming quality barriers in machine and human translation and in language technologies.

To this end, on the premises that translation quality assessment today is hampered by subjectivity, poor usability, and the inability of current models, especially as to effectively dealing with machine translation, a project has been launched to develop a flexible, modern, and open metrics.

The Multidimensional Quality Metrics project goal is to build next-generation metrics based on the following principles:

  • Adaptability and flexibility for different project type, users, and workflows
  • Fairness to all stakeholders
  • Applicability to source and target to promote integration of the document-production lifecycle
  • Suitability for both human and machine translation workflows
  • Comparability across domains, projects, and language pairs
  • Standardization upon existing ISO specifications and popular models
  • Usability to ensure ease of use and uptake
  • Inclusivity through a community-based effort including translators, companies, researchers, and tech-doc creators.

As a starting point for the MQM project, the following definition of translation quality by Alan Melby has been used: “A quality translation (1) demonstrates required accuracy and fluency (2) for the audience and purpose and (3) complies with all other negotiated specifications, taking into account end-user needs.”

Keywords here are ‘required’ and ‘negotiation specifications’.

In ISO 8402 standard of 1986, “Quality management and quality assurance – Vocabulary”, now withdrawn, quality was defined as “the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs.”

In the third edition of ISO 9000 series of standards of 2005, “Quality management systems — Fundamentals and vocabulary” quality was then defined (3.1.1) as the “degree to which a set of inherent characteristics fulfills requirements”, where requirement is defined (3.1.2) as the “need or expectation that is stated, generally implied or obligatory.”

Also, ASTM F 2575 — 6, in 5.1.1, defines quality as “the degree to which the characteristics of a translation fulfill the requirements of the agreed-upon specifications.”

There seems to be no need, then, for a distinct definition of quality as to translation. Therefore, the new approach admittedly followed for MQM looks very much like the same old ones, that is to reinvent the wheel rather than trying something really new and innovative.

The supposed need for a definition of quality specifically for the translation industry re-affirms the view of translation as an independent, distinctive, and somewhat outstanding and exclusive activity.

Not surprisingly, according to GALA’s representative at ISO TC 37, Demid Tishin, ISO 9001 was a too broad quality management standard that did not specify translation services and was not hugely popular.

MQM are intended as a framework based on ISO technical specification 11669 to provide translation requirements and select the issues for specific needs in quality assessment.

As a matter of fact, despite showing this model as analytical, it is still based on an error-catching rather approach than on an error-preventing approach.

Just like ASTM F 2575 — 6 and (although in a less strict view) EN 15038 and CAN/CGSB-131.10-2008, ISO/TS 11669 provides guidance concerning best practices for all the phases of a translation project. The rationale for translation project specifications is to be attached to a legally binding contract or, in the absence of it, to a purchase order (or any other document supporting the request to define the work to be done).

The provision of project specifications is then the first best practice to implement, since the quality of a translation can supposedly be determined by the degree of compliance to the predetermined specifications. In this sense, project specifications are also the benchmark for qualitative and quantitative assessments.

Like EN 15038 and CAN/CGSB-131.10-2008, ISO/TS 11669 does not provide any procedures for quantitative measurement of the quality of a translation product, thus establishing the rationale for MQM.

The real news in ISO/TS 11669 are translation parameters, intended as key factors, activities, elements and attributes of a given project used for creating specifications. However, the long listing of translation parameters is a surreptitious way to levy vague and blurry translation quality assessment criteria, which are traditionally subjective.

Also, since quality is defined as the degree to which the translation product conforms to the project specifications, and since no guidance is given for qualitative assessment, register should not be a parameter, as its compliance to requirements is highly subjective.

Again, the distinctions between ‘overt’ and ‘covert’ translation, and ‘foreignized’ and ‘domesticated’, and the insistence on ‘register’ are all underhand endorsements of the archetypal information asymmetry between requesters and providers. They all assume that the requester has the same knowledge — and/or interest — as the provider, while they are all elements of the typical academic translation education. Language and translation competence are perfect examples.

In this scenario, it is not surprising that preventing errors is outside the scope of the metrics, as well as in the referenced specification. Catching (and measuring) errors is still the main, if not the only, focus, while reducing error categories  does not represent a step forward.

QT LaunchPad’s metrics break quality into three areas:

  1. Fluency
  2. Accuracy
  3. Adequacy.

None of them is immune from subjectivity, while objective measurability and thus its repeatability is at the very heart of quality assurance. In mass production, errors are prevented, or at least reduced, using objective specifications and sampling techniques for inspection. Both are absent from ISO/TS 11669 and MQM. For further reading, see “Quality assessment and economic sustainability of translation.”

The formula proposed to calculate the MQM quality score, that is the overall rating of quality, also seems flawed.

Q = (Atr + At – As) + (Ft – Fs)

Where the first three values give the accuracy score and the last two  give the fluency score. The accuracy score is calculated as the sum of overall transfer accuracy (Atr), target accuracy (At) and source accuracy (As), while the fluency score is calculated as the sum of fluency in target (Ft) and source text (Fs).

This could lead to a paradoxical assessment of a translation better than the original. Besides signaling a non-compliance, this could be a non-sense in an economic perspective, with the customer having a wrong perception over the money spent and the vendor providing more for less. The value could also be prone to high subjectivity, while leaving the customer to be the final judge, as it should be, but eliminating the need for any TQA.

Within the QT LaunchPad’s MQM project, an application has been developed for the translate5 web-based translation editor.

A long road aheadAlthough this application is still in a beta stage, no features are available for error tracking from assessment reports, with no intelligent error fixing being possible. Also, with no specification like ISO/TS 11669 for the preparation of texts, within any assessment of source texts any error prevention action will be arbitrary. Finally, there are no means to assess error fixing, which could be extremely useful for PEMT tasks.

However, QT LaunchPad’s metrics will positively be open and available to the public at no charge, which hopefully leaves ample margins of short-term improvements.

In conclusion, if in the past translators were unwilling to be involved, as differences were too many and too wide, for  such public-funded projects to be effective and factual, the need for collaboration and understanding of all the parties involved in translation projects is now higher than ever.

There’s a long road ahead to go.

Author: Luigi Muzii

Luigi Muzii