Measuring for success: Goals, performances, and outcomes

This presentation was scheduled to be held at tcworld 2017 in Stuttgart on October 26, 2017.

Introduction

Every business should measure performances against goals to substantiate its existence and justify paychecks on solid arguments that customers can understand.

This paper is designed to provide an introduction to KPIs and their value for businesses. Suggestions are given about developing vertical KPIs that can be understood by customers.

Anything can be measured

A costly myth permeates many organizations today, that certain things cannot be measured. Actually, measuring is no rocket science, and anything can be measured. In fact, if a thing can be observed, observations will always tell something that was unknown before, because anything that can be observed lends itself to some type of measurement. So, any measuring effort, however vague, leads at least to learn what to measure, and how and eventually to know more about the thing being measured. On the contrary, when something is believed to be unmeasurable, no measuring effort is even considered. This leads to poorly-informed decisions, a higher chance of error and, eventually, to money waste.

The Fermi Problem

Measuring helps reduce uncertainty, and Enrico Fermi was known for his ability to make good ballpark calculations with little or no actual data and for teaching his students how to approximate unknown quantities and solve problems where confirming the results would be hard. The best-known example was Fermi’s estimate of the number of piano tuners in Chicago.

Fermi would start by asking students to make seemingly easy estimates such as the current population of Chicago, the average number of people per household, the share of households with a piano that would be tuned regularly, the required frequency of tuning, how many pianos a tuner could tune in a day, and how many days a year the turner works.

The result would be computed:

Tuners in Chicago  =  Population/people per household
• percentage of households with tuned pianos
• tunings per year ÷
(tunings per tuner per day • workdays per year)

Why measure?

Fermi’s method helps estimators approximate an uncertain quantity and provides a basis for seeing where the uncertainty came from to possibly point toward a measurement that would reduce it.

If an object seems unmeasurable and yet there is a reason for measurement, it is because measuring is expected to give certain results that are meant to support a decision. Indeed, an object may seem unmeasurable simply because the rationale for measurement has not been clearly defined. Therefore, it is essential to define goals and make observation easier, by possibly breaking down the object into small observation blocks.

Essentially, the reasons for measuring are:

  1. To know something unknown;
  2. To learn what and how to measure;
  3. To reduce uncertainty and the consequent risk of wasting money.

Risks

“In God we trust. All others must bring data.” is a famous quote attributed to William Edwards Deming, the father of modern quality management. It calls for the importance for any business of understanding how it is performing and, more importantly, identifying ways to improve it through the data it gets created every day. In other words, it means that, when coming to business, no one should trust anything that is not backed by data.

Turning data into useful information to generate better decisions might prove less hard than expected, while in business, uncertainty (a situation involving imperfect and/or unknown information) generates risk, that is a potential loss.

Process management

In the latter part of the twentieth century, with the goal to meet customer requirements profitably, companies all over the world started to adopt a process-oriented approach to run their business by focusing on how work is done.

In this approach, processes are designed and implemented as a control system to pursue goals while complying with laws and regulations, meeting operational needs, and managing risks. The end results of every stage are labeled as outcomes. The capacity to attain these outcomes through preset known standards is labeled as performance.

Traditionally, businesses focus on performances as performances can be measured against goals, while customers are interested in outcomes as the product or service they receive and pay for. Unfortunately, outcome goals cannot be controlled.

Performance goal Outcome goal
Run the 100m race in 10” Win first place in the contest
Tackle your opponent out Win the rebound
Sprint after ball comes into play Get to the ball first and control it

Between the 1970s and the 1980s, with the progressive spreading of total quality management (TQM) principles, the concept of continual improvement was taking more and more ground to attain the ultimate consecration in the early 2000s with the global adoption of ISO 9001. In ISO 9001, since the 2000 issue, a specific clause has been dedicated to the requirement for continual improvement defined as the recurring activity to increase the ability to fulfill requirements.

28 years after the first edition, a new edition of ISO 9001 was released in 2015 that focuses on performance and combines the process approach with risk management to make the quality management system a preventive tool helping continual improvement. Measuring performance against organizational goals has become crucial to identify opportunities for continual improvement and risk reduction by assessing processes, streamlining work, improving efficiency and reducing waste.

Also, a major issue in the service industry comes from information asymmetry. Buyers and sellers have different information about the object(s) in a transaction, and this prevents buyers from assessing the value of outcomes through examination before sale is made. Buyers are often incapable of getting the complexity of tasks in professional services and perceive a risk ensuing from the escalation of costs from possible poor workmanship of delivery and rework. The result is an imbalance of power and a pressure on price to mitigate risks. To overcome information asymmetry, vendors can use measures for signaling to convey some information about themselves to buyers that might be positively correlated with capabilities.

First steps

Before making any attempt at measurement, the following questions must be answered:

  1. What decision the measurement is supposed to support?
  2. What is the definition of the thing being measured?
  3. How does the thing to be measured matter to the decision?
  4. What is the current level of uncertainty?
  5. What is the value of additional information?

When an urgency for measuring something is felt, this is usually because a decision is going to be made and the measure is expected to support it. If this something seems unmeasurable, it may be because a certain measure is expected. Since this measurement is meant to support a decision, the object of the measurement, the relevant decision, and how the measurement is going to affect the decision must be clearly known and defined. Breaking down the object of measurement in small blocks that can be more easily defined may help observation and measuring.

An urgency for measuring may also come from uncertainty and be meant to reduce the ensuing risk. In this respect, measurements do not have necessarily to express exact quantities. Just think of garment sizes, hotel ratings, rankings, etc. In any case, by reducing uncertainty, and thus risk, measurements are supposed to add value, and understating this value is crucial to assess their impact.

To categorize variables and the accuracy of measurements, scales are used, nominal, ordinal, interval and ratio. Nominal scales do not express quantities, and boolean statements are often used: yes/not, true/false, good/bad, etc. Ordinal scales consist of a spectrum of values, allowing for ranking, to say whether one value is more than another, but not by how much. Interval scales express a numerical difference between two values. Ratio scales are interval scales with a zero position to indicate absence. Most measurements in the physical sciences and engineering are based on ratio scales.

What and how to measure

Measurements should be taken on the core task of the key areas of any business.

A typical measurement is price, which is affected by the cost of service. Price pertains to the sales area, while the cost of service pertains to operations departments. Measurements taken in sales and operations relate to customer satisfaction, which, in turn, affects the company’s reputation and sales. Vendor rating is a typical measurement for procurement departments affecting resource development, which should be measured too. Vendor rating and resource development also affect quality and productivity, which should be measured too as they are affected by vendor capability and capacity.

In most cases, though, the simplest and typical measurements involve three major areas, processes, finance, and personnel. Each measurement in any of these areas depend on variables that respond to different measuring approaches. Price is a function of the cost of service, but also of the shipping capacity and project management effectiveness. The quality of service depends on analytics, and on the investments for any corrective measures to ensure continual improvement. Customer service depends on the quality of service and on maintenance costs, the latter being a component of the cost of service. Flexibility depend on the planning ability that allows an organization to be resilient to peaks of demand, and planning is an essential element in project management. Finally, if creditworthiness depends on cash flow and, ultimately, from efficiency, reputation depends on staff preparation, which in turns is a product of resource development.

To make its efforts more efficient, cost-effective, and insightful in the customer’s perspective, every organization’s measurement policy should operate on baselines, thresholds and benchmarks. Baselines are the starting point for any comparison and are especially useful in pricing. Thresholds are the base for negotiations on performances, with AQLs (Acceptance Quality Levels) being a perfect example. Benchmarks express average performances in industry against which measurements should be made.

Buyers and vendors have different expectations on measurements. Buyers are interested in assessing a vendor’s capability while vendors are interested in the buyer’s reliability. However, buyers and vendors share an interest in service levels, so any measures must be intelligible to both.

So, when measuring against goals, the first step consists in identifying performances to measure, and in choosing metrics to express and correlate results. Measurements must be made on standardized samples and tools to assess compliance according to pre-specified benchmarks. For example, the speedometer in a car is a typical SPC (Statistical Process Control) chart, with minimum and maximum limits corresponding to nonconformance. A value on the chart can be used as baseline or threshold. The average speed to get to destination in a predetermined time is a baseline. Maximum speed (as on urban roads) and minimum speed (as on highways) are thresholds. In the same context, the speed at which a car and its consumption are outcome indicator, and the ratio between speed and consumption is a performance indicator, expressing how economically the car is being driven.

Today’s car dashboards are full of counter and gauges. Control units can provide instant outcome and performance indicators such as speed, average speed, consumption (in l/km) and mileage or the expected number of kilometers to drive with the available gas in the tank at current speed. Some counters may even seem almost useless. In fact, it is always up to the driver, to decide when to press or release the speed pedal, when to change gear, etc. that is to make the decisions those measurements are meant to support.

KPIs (Key Performance Indicators)

Back to the areas any organization should focus on, there are some more basic measurements:

Processes Finance Personnel
  • Efficiency
    • Overhead
    • Projects closures
    • Complaints
      • Unresolved complaints
    • Administration
  • Quality
  • Automation
  • Sales
    • Successful proposals
    • New customers
    • Customer profitability
    • Customer loyalty
  • Revenues
    • Cash flow
  • Profits
    • ROI
  • Staff
    • Number
    • Turnover
  • Development
    • Empowerment

These measurements are performance indicators that help an organization understand how well it is performing and make decisions accordingly. Those relating to strategic goals are key performance indicators (KPIs.)

For example, if quality is a strategic goal, a process quality indicator may be computed out of six performance indicators expressing the organization’s capability:

  • The Capacity Utilization Ratio (CUR);
  • The DIFOT (Delivery In-Full, On-Time) rate;
  • The FPY (First Pass Yield) rate;
  • The Order Fulfillment Cycle Time (OFCT);
  • The Rework Level;
  • The rate of customer complaints solved.

CUR expresses the output produced in a given time-frame and reflects the way in and the extent to which an organization uses its installed productive capacity. The difference to 100% indicates room to improvement without incurring costs of increasing capacity, while a low value highlights serious process inefficiency.

DIFOT express the ability of a business to fulfil orders and meet customer expectations and provides a measure of the effectiveness and efficiency of processes and supply chain, conveying a measure of delivery reliability.

FPY expresses the percentage of units coming out of a process with no rework and provides a measure of process effectiveness.

OFCT expresses the average time taken to source, make and deliver a product or service from order to customer receipt. It represents the total “time waiting” experienced, and provides a measure of an organization’s delivery capacity in an end to end process.

The rework level gives the percentage of items inspected requiring rework and provides a measure of an organization’s operational efficiency at delivering the product as specified by the customer without further correction, alteration or revision.

Finally, complaints allow an organization to know what is wrong with its products or services and how to improve them, highlight any weak links within the organization, and tell what is important to customers or give ideas for new products and services. Most customers do not complain and just take their business elsewhere. Therefore, adequate complaint handling, up to the solution of the problem and its communication to the public, could be crucial and the organization’s capability in this respect is expressed with three indicators:

The number of complaints received from customers divided by the total number of items delivered over the same period of time;

The number of complaints solved divided by the total number of complaints received from customers;

The total number of hours required to successfully resolve a customer complaint, from the time the complaint is submitted until when the complaint is resolved and closed divided by the total number of hours worked.

Data

In a world of data, when measuring seems hard it may be due to the lack of specific tools for certain vertical domains.

As indicators are meant to help perceive differences, improvements, or developments, they can be quantitative or qualitative. Quantitative indicators can be expressed with a number, while qualitative indicators cannot. As quantitative indicators just need mechanical methods that are theoretically expected to give the same results, they are seen as objective. On the contrary, qualitative indicators depict a perception based on experience and are seen as subjective and, as such, unreliable. Anyway, qualitative indicators can play an important role in identifying constraints, thus in understanding the perspectives of stakeholders. Quantitative indicators are used to measure outcomes, while qualitative indicators are used for judgements. Approximations are inevitable in any case and, as quantitative indicators are easier to understand and manipulate, they must be the base for qualitative indicators.

For example, error-based metrics like Six Sigma may be useful for gathering insights on the stability and predictability of process outcomes. However, since Six Sigma was originally tailored for highly standardized production environment, error definition and error gravity are pivotal. As an analogy, consider the application of Six Sigma to soccer. If a defect is a score from opponents, a goalkeeper in a level-6 team playing 50 games in a season and facing 50 shots per game would concede one goal every 147 years (0.000034 percent of errors.)

You can design and implement your own indicators. In this case, a few basic rules should be followed:

  1. Identify the strategic goal(s) for each indicator;
  2. State the question(s) that the indicator is meant to answer;
  3. Specify how each indicator will be used and shall not be used;
  4. Identify and describe which data should be collected and used, and how;
  5. Specify the assessment criteria (qualitative or quantitative) and the associated scale;
  6. Identify baseline, benchmark, and thresholds for each indicator.

Also, to be comprehensive and effective, any KPI should show four features:

  • Current value, the current performance;
  • Plan value, the expected performance.
  • Deviation value, the difference between the current and the expected performance.
  • Trend value, an estimation based on historical data.

Effective KPIs are all about data, especially project data. The number of indicators should be based on available and collectible data; usually 5-10 is enough for gauge charts. Most of the data to “feed” a KPI dashboard can be retrieved from job tickets. The more detailed a job ticket is the better for tracking the project management effort and computing performance indicators. In any case, clean raw data should be used, possibly after extraction. This data can then be processed using one of the many free Excel templates available, especially if no (customizable) KPI tool is available on the ERP platform of choice.

Anyway, even in measuring, a little common sense goes a long way.