qualitative measures

I understand the lure of a set of numbers that tell you how you’re tracking, but putting metrics in place to measure success is a move fraught with danger.

Perverse incentives

One of the key challenges is perverse incentives. For every metric you come up with, there’s a way you can improve that metric by going completely against the spirit of what you’re trying to achieve.

The most popular example of this is the Hanoi government trying to reduce the number of rats by offering a bounty for dead rats. What happened? People started breeding rats.

Or closer to home, how about the call centre I’m not allowed to name who had a KPI around calls being answered within fifteen minutes. Staff quickly figured out that if the call had been on hold for fifteen minutes, it was better for their stats not to answer it at all. So everyone they spoke to had been on hold less than fifteen minutes (yay, bonuses all round!) but their abandoned call rate skyrocketed.

I could list countless examples of this off the top of my head, and you could find more yourself just by googling it. The moment you put a metric in place, you open yourself up to people ruining it, because they want to meet a target not the intent of the target.

Could we measure the outcome, not the output?

Maybe. Firstly you need to be clear on what the outcome is, and then you need to have ways to measure it.

It’s not easy. For example: think about building a block of apartments, where the outcome is creating a home (not just a place to live) where people feel comfortable and a part of their community. Traditional measures of success for apartments might be:

• How quickly they sell.
• The price they sell for (comparative to build cost).
• The price they rent for.
• Winning a design award.

They’re all very tangible things you could put a number on and compare to other apartment blocks in the area. But they’re all measuring the output; the physical block of apartments. None of them measure the outcome; a home.

You can get measures that are more geared towards outcomes, such as:

• How long people stay.
• How many of their neighbours they know.
• How much time / money they spend locally.
• Number of owner/occupiers versus renters.
• Satisfaction with the property over time.

As you’ll see from these, outcomes measures are likely to be longer term, and on their own they won’t tell you if you’ve really created a home. You’ll still need to do some work — qualitative research, not just quantitative — to understand if you’ve succeeded.

And sadly, most people involved in designing an apartment block won’t ever be around to do this. Once the design is finished they move on, often before it’s even built.

What if evaluation was about learning, not measuring?

Another option is that we turn evaluation on its head and use metrics as a way to learn instead of to measure.

If the call centre with the fifteen minute rule adopted this, they’d end up with a completely different situation. If they’re not rewarding or punishing anyone for the target, but are just monitoring it along with a whole lot of other numbers, they might learn:

• What days or times they have the longest hold times.
• What sort of things trigger longer hold times.
• If there are early warning signs for when hold times are about to blow out.

Knowing this, they could explore ways of dealing with hold times differently; predict when they need more staff or institute a call back system or something else completely.

This works well for an ongoing service like a call centre, but it’s a bit harder for a one-off build like the block of apartments. Having said that, if the architect used each building as a learning exercise and made a point of going back to understand what it’s like to actually live in the places they built, I have no doubt they’d create better apartments next time around.

Conclusion

Sadly, we’re not getting away from metrics any time soon. Business cases tend to demand quantifiable things that are easy to measure; and people like to be able to point to a statistic and say “look, it’s working!” (even if it’s not).

Push back where you can, but if all else fails and metrics are going to be used to measure success, try:

• Thinking about the outcome you’re trying to achieve, not just the output, and all the ways you might measure that. Include some of those as part of the plan.
• Building your evaluation plan to be about learning, not just about measuring.

Both of those things are almost certainly going to involve both qualitative and quantitative measures, and will help you build a more holistic picture of what’s going on.

Further reading

Check out the terrific Wikipedia page for more examples of perverse incentives: https://en.wikipedia.org/wiki/Perverse_incentive

Also this scholarly article on Academic Research in the 21st Century and all the ways they got incentives wrong: https://www.liebertpub.com/doi/10.1089/ees.2016.0223

And this article on Perverse Incentives: The Limits of Metric-Based Evaluation: https://www.ieseinsight.com/doc.aspx?id=1907&idioma=2