Measure for Measure

Performance indicatorMy favorite quotation on quality comes from Peter Drucker: “Quality in a product or service is not what the supplier puts in. It is what the customer gets out and is willing to pay for. […] Customers pay only for what is of use to them and gives them value. Nothing else constitutes quality.

This means that what constitutes “quality” largely depends on the buyer’s perspective.

Another of my favorite statements on quality comes from Philip Crosby: “Do it right the first time.”

In his 1979 best seller, Quality Is Free, Crosby also stated that “Quality is free. It’s not a gift, but it is free. What costs money are the un-quality things—all the actions that involve not doing jobs right the first time.” In a quality system view, “every time” should be appended.

The highest quality is the lowest cost. Cost, not price. Because prices are reasonably similar, the highest quality at the lowest cost will make economies of scale possible and will increase productivity.

Since quality depends on the buyer’s expectations, meeting requirements is vital. To meet requirements and scale production, performance must be measured. To measure performance, metrics are needed.

This partly explains why, in the translation industry, any consideration on quality is made with regard to the cost of poor quality (COPQ), meaning the cost that would disappear if systems, processes, and products were perfect. Unfortunately, this is still not a valid measure (what is perfect?), as it can only be obtained afterwards and is not repeatable, let alone predictable.

Immeasurability of Translation Quality

Much has been written on the immeasurability of translation quality, which is mostly due to the typical subjectivity involved in assessment. Translation quality cannot be measured by the number of translation errors alone. Even if a translation does not contain any blatant errors (any material mistakes that could lead to a substantial COPQ for any risks related to possible damages), it can still be of poor quality — at least for someone.

In fact, translation trainees are still taught that translation errors are just one indicator of poor quality, and that a translation cannot be judged by the number of mistakes only. Style, tone and register should also be taken into account because a poor style, the inappropriate tone, or the wrong register can make it a bad translation, even if it contains no material mistakes.

This is probably the biggest reason why translation as a job is not taken seriously, even though the general opinion is that the translation industry is as worth as the bicycle industry and suffering no crisis.

Service or Product

Another argument relates to production and the ongoing equivocal dispute of service vs. product. Every translation is a product resulting of a process, with a combination of tangible and intangible attributes that a seller offers a buyer.

Translation is not a service per se, even though parts of the production process can be offered as standalone services. Typical services are intangible and insubstantial, can be (re)sold or owned by somebody, but cannot be turned over from the service provider to the service consumer. Services are also perishable and inseparable from provider. A translation is a product. Even though, just like services, each translation is unique and one-time generated. It can be one-time rendered and consumed, and can never be exactly repeated because circumstances and conditions are always different, even when the same service consumer requests the same service and location, and configurations and assigned resources are the same.

This is so vastly acknowledged that, in issue 6 (02/2013) of Languages and Translation, the EU DGT magazine, Professor Philipp Koehn, writes: “Translation is unpredictable. If you give a text in a foreign language to a group of translators, you can be sure that each will come up with a different translation. That is true even if the text consists only of a single sentence with a few words.

This uniqueness is where the confusion product vs. service stems from, and is a major impediment in translation quality assessment, starting with the fact that contrastive analysis is still the preferred (and possibly the only) way to perform it, requiring a substantial knowledge of the language pair in a translation.

Reducing Unpredictability

Machine translation can reduce these traits of uniqueness and unpredictability of translation, thus being the very first step towards an industrial approach to translation.

And yet, even professor Koehn warns that “the costly task of producing translations that are of publishable quality still requires professional human translators for the foreseeable future.”

But what does “publishable quality” mean? Publishing is up to the customer, it is in his prerogatives. Then, again, quality is what the customers define as such.

Quality Is a Given

What makes a translator different from another? Definitely not “quality” or something like that, at least in the broad sense translators usually accord to the term. It is the customer’s choice. It does not depend on the intrinsic features of a product, but on the extent to which the product meets the initial requirements.

Therefore, quality does not sell in translation: it is a prerequisite for existence on market, it is a given.

This is no news: Renato Beninatto has been saying the same for years, at least since 2007, and anyway long before Marcela Reyes, the last in time. Even though a traditional approach can be typically found among language professionals, having been educated as linguists or translators, it is quite familiar also to LSPs.

Not surprisingly, quality is the unique selling proposition of the entire translation industry helping make this a market for lemon, in a lose-lose situation. Quality as a signal is useless and pushes customers towards price as the main (when not the only) differentiator.

LSPs and translators can differentiate by providing outstanding services along with the product and signaling to customers using indicators that customers can understand.


KPIsIn an article for the Multilingual Magazine #57 Supplement of July/August 2003 titled Establishing Key Performance Indicators for Localization, Don DePalma wrote: “Localization does not cost a lot of money.

This statement would blatantly contradict the vulgate that companies are not willing to spend in translation, but then DePalma added: “Interviewees spent between one-quarter of one percent and 2.5% of their non-US, non-Anglophone-market revenue per year to localize product documentation, user interfaces, Web sites and service-related materials for six to ten markets.” This is a confirmation of the reluctance to spend on a task which apparently is not perceived as valuable and important and, therefore, not considered for an investment. Not only in the U.S.

Further on in his article, DePalma added that interviewees “found that no one could really tell [them] what it actually cost to create the original owner’s manuals, marketing materials and Web sites that get localized” and concluded that it was “time for localization practitioners and suppliers to communicate their value in business terms.

Finally, DePalma argued that “LSPs should measure performance against goals and that practitioners both inside companies and at the firms that supply their localization service and technology needs often justify their existence and paychecks to each other, each building on the same arguments and data. Few appeal in terms that budget owners can understand, thus compounding the irony of people responsible for communication being unable to communicate.

That was ten years ago. Has anything changed since then?

On January 25, 2013 Common Sense Advisory returned to the topic with an 8-page research aiming at showing how to turn a company mission and corporate goals into Key Performance Indicators (KPIs) and develop them in seven steps.

Key Performance Indicators (KPI): Developing, Implementing, and Using Winning KPIs is a 256-page book devoted to KPIs. Its author, David Parmenter, states: “Show me a company who thinks they have KPIs, which are measured monthly and quarterly, and I will show you measures that do not create change, alignment and growth and have never been KPIs.” Key Performance Indicators (KPI): The 75 measures every manager needs to know is another 376-page book on KPIs by Bernard Marr, who does not claim to be “the King of KPIs,” but who is nonetheless an acknowledged authority in this field.

Developing KPIs

A performance measurement is a comparison of actual returns against a pre-specified benchmark. A performance metric is a type of measurement used to quantify the performance of some component of an organization.

KPIs are metrics to measure the success of a business. Developing a set of effective KPIs is challenging. In fact, as Parmenter states in his book, very few companies have explored what a KPI actually is and many are working with the wrong measures.

In his book, Parmenter indicates seven characteristics of KPIs. KPIs:

  1. Are nonfinancial measures
  2. Are measured frequently
  3. Are acted on by the CEO and senior management team
  4. Clearly indicate what action is required by staff
  5. Are measures that tie responsibility down to a team
  6. Have a significant impact
  7. Encourage appropriate action.

Since KPIs should tell what to do to increase performance dramatically, Parmenter also suggests no more than 10 KPIs and presents a 12-step model to develop and use them.

On the other hand, in his book, Marr lists 75 measures, and groups them under six perspectives:

  1. Financial
  2. Customer
  3. Marketing and sales
  4. Operational processes and supply chain
  5. Employee
  6. Corporate social responsibility

Since money is always a critical component of any business decision, the primary focus of customers is still to get the best price from their vendors, even though every customer knows that the lowest price doesn’t necessarily buy the best solution. This is especially true for translation, because of the typical information asymmetry of the industry.

KPIs can then be used as a showcase for customers as well as a tool to measure the progress of the business toward its goals. In the first case, KPIs can be offered to companies with significant translation requirements to allow them to assess their vendor’s capabilities and know how their translation budget is getting spent. KPIs can also be reported to customers to show the vendor’s effort towards them.

On the other hand, LSPs can use KPIs to measure process effectiveness, assess employee performance, customer satisfaction and productivity.

Which KPIs can be developed that can be understood by customers, and how?

Following Parmenter’s and Marr’s approaches together, 10 KPIs can be identified for the typical LSP. EVA, the customer retention rate, the customer satisfaction index, and the customer profitability score are all possible indicators in the customer’s perspective.

CUR, DIFOT, the human capital value added, the employee satisfaction index, the salary competitiveness ratio, and the time to hire could be indicators from the LSP’s perspective.

Once again, using quality as an indicator is misleading, even from a management standpoint. KPIs are only statistically valid if there are enough project data points, exactly what no LSP or translation professional is still capable of producing, because quality standards for the translation industry are based on obsolete models.

According to Nataly Kelly, “investors often come to Common Sense Advisory when conducting their due diligence on the language services market [and] they are usually fascinated by the prospect of bringing better technology and automation to a market worth more than $33 billion that currently consists primarily of human-delivered services. However, many funders fail to seek out information in order to understand the nuances of this market.” And we are back at the beginning.

Unfortunately for investors, merely having a great technology is no guarantee,” Kelly concludes: “We’ve been urging LSPs to showcase translators and bring them out of hiding for years.” But that’s KPIs from the LSP’s perspective above.

Suggestions are welcome for more KPIs.