qualitative measures Archives

8 April 2025

Generative AI is unethical, and counterproductive

Alexandra Almond Design philosophy measurements, metrics, qualitative measures, quantitative measures

AI is pervasive; you have to make a concerted effort not to use it these days. And while many people have jumped on the bandwagon willingly, others are being taken along for the ride without thinking about the implications.

Those implications are critical and I want you to at least consider them. I’m against the use of generative AI for two main reasons[i].

It’s theft.

The large language models (LLMs) that form the basis of current AI models have been trained on the pirated work of professional authors. I’m talking about all the big technologies; ChatGPT as well as the models developed by Google, Meta, Amazon and Microsoft.

We suspected this was happening, because if you asked ChatGPT to write something in the style of a published author, it would be able to do so. Maybe not very well, but with enough recognisable traits that you could see the resemblance.

But we now have proof, following a recent US court case where Meta was forced to reveal their sources. A groundbreaking article published by The Atlantic[ii] last month linked to the pirate database Meta used to steal 7.5 million books. It also revealed that Meta had investigated the option of signing licensing agreements with authors and publishers but decided that would be “incredibly slow” and “unreasonably expensive”.

Authors have not given permission for their work to be used for this purpose, nor have they been compensated.

Yes, I have a stake in the game here. My novel THOROUGHLY DISENCHANTED is in the LibGen database that was used by Meta, and I’m furious about it. These are some of the world’s richest technology companies, who have elected to steal what they need rather than pay for it. That’s unethical and illegal and just plain wrong.

It won’t help in long run

Generative AI is a shortcut. I love shortcuts when the end goal is more important than the journey. For example, if I could have clean sheets every day without ever having to wash them, I’m there. There’s no intrinsic value in me changing my sheets and putting them through the washing machine, and in fact it’s bad for the environment[iii].

However, many of the ways I see people using Generative AI are in moments where you will lose something by not doing it yourself. For example:

Designers analysing research findings in a design project. See my previous post about synthesis[iv] where I argue it’s the act of sorting, not what categories you sort the data into, that’s important. The whole purpose of synthesis is to get your brain working so that you start to think about something in a way you haven’t thought about it before. Getting a computer to do that analysis for you won’t help your brain process information, and won’t get you thinking the way you need to move forward in a meaningful way.
Authors writing a synopsis, or plotting their novel, or actually doing anything. Bastardising the words of Chuck Jones[v] to apply to writers; every writer has thousands of bad stories in them and the only way to get rid of them is to write them out. You don’t learn anything by getting someone (something) else to do the hard work for you. Please see this excellent blog post by KJ Charles[vi] for a better description of why using AI to shortcut your writing process is counterproductive.

Absolutely you can get a quicker output for a particular task by using AI, but the outcome will be worse and you will have failed to learn anything from the process. That means you stop developing, stop getting better, stop learning how to improve.

I have never used ChatGPT or any of the other generative AI tools. I downgraded my Microsoft subscription so it doesn’t include CoPilot, and I switched my search engine to Ecosia to avoid the automatic AI results that google comes up with at the top of every search. I am also removing myself from all Meta social media sites.

This is a key ethical dilemma for our time. Please think about the implications next time you go to use ChatGPT (or any of the others) as a shortcut.

Photo credit: Adrian Pope https://fusionchat.ai/news/meta-faces-backlash-authors-protest-in-london

[i] There’s also the fact that the outputs from generative AI aren’t very good (factually inaccurate, often incoherent and usually quite obviously not created by a human) but that’s not an argument that will stand the test of time. They will get better as they learn, it’s what they’re designed to do.

[ii] https://www.theatlantic.com/technology/archive/2025/03/libgen-meta-openai/682093/

[iii] AI also has an environment problem, but I’m not confident about the facts to put that as my third reason not to use it. Here’s one article about that side of things: https://www.unep.org/news-and-stories/story/ai-has-environmental-problem-heres-what-world-can-do-about

[iv] https://alexandraalmond.com.au/synthesis-it-doesnt-matter-how-you-categorise-your-post-it-notes/

[v] https://quotefancy.com/chuck-jones-quotes

[vi] https://kjcharleswriter.com/2024/08/19/weak-ankles-and-ai-an-extended-metaphor/

30 June 2024

Metrics are bad (sometimes)

Alexandra Almond Design philosophy measurements, metrics, qualitative measures, quantitative measures

I understand the lure of a set of numbers that tell you how you’re tracking, but putting metrics in place to measure success is a move fraught with danger.

Perverse incentives

One of the key challenges is perverse incentives. For every metric you come up with, there’s a way you can improve that metric by going completely against the spirit of what you’re trying to achieve.

The most popular example of this is the Hanoi government trying to reduce the number of rats by offering a bounty for dead rats. What happened? People started breeding rats.

Or closer to home, how about the call centre I’m not allowed to name who had a KPI around calls being answered within fifteen minutes. Staff quickly figured out that if the call had been on hold for fifteen minutes, it was better for their stats not to answer it at all. So everyone they spoke to had been on hold less than fifteen minutes (yay, bonuses all round!) but their abandoned call rate skyrocketed.

I could list countless examples of this off the top of my head, and you could find more yourself just by googling it. The moment you put a metric in place, you open yourself up to people ruining it, because they want to meet a target not the intent of the target.

Could we measure the outcome, not the output?

Maybe. Firstly you need to be clear on what the outcome is, and then you need to have ways to measure it.

It’s not easy. For example: think about building a block of apartments, where the outcome is creating a home (not just a place to live) where people feel comfortable and a part of their community. Traditional measures of success for apartments might be:

How quickly they sell.
The price they sell for (comparative to build cost).
The price they rent for.
Winning a design award.

They’re all very tangible things you could put a number on and compare to other apartment blocks in the area. But they’re all measuring the output; the physical block of apartments. None of them measure the outcome; a home.

You can get measures that are more geared towards outcomes, such as:

How long people stay.
How many of their neighbours they know.
How much time / money they spend locally.
Number of owner/occupiers versus renters.
Satisfaction with the property over time.

As you’ll see from these, outcomes measures are likely to be longer term, and on their own they won’t tell you if you’ve really created a home. You’ll still need to do some work — qualitative research, not just quantitative — to understand if you’ve succeeded.

And sadly, most people involved in designing an apartment block won’t ever be around to do this. Once the design is finished they move on, often before it’s even built.

What if evaluation was about learning, not measuring?

Another option is that we turn evaluation on its head and use metrics as a way to learn instead of to measure.

If the call centre with the fifteen minute rule adopted this, they’d end up with a completely different situation. If they’re not rewarding or punishing anyone for the target, but are just monitoring it along with a whole lot of other numbers, they might learn:

What days or times they have the longest hold times.
What sort of things trigger longer hold times.
If there are early warning signs for when hold times are about to blow out.

Knowing this, they could explore ways of dealing with hold times differently; predict when they need more staff or institute a call back system or something else completely.

This works well for an ongoing service like a call centre, but it’s a bit harder for a one-off build like the block of apartments. Having said that, if the architect used each building as a learning exercise and made a point of going back to understand what it’s like to actually live in the places they built, I have no doubt they’d create better apartments next time around.

Conclusion

Sadly, we’re not getting away from metrics any time soon. Business cases tend to demand quantifiable things that are easy to measure; and people like to be able to point to a statistic and say “look, it’s working!” (even if it’s not).

Push back where you can, but if all else fails and metrics are going to be used to measure success, try:

Thinking about the outcome you’re trying to achieve, not just the output, and all the ways you might measure that. Include some of those as part of the plan.
Building your evaluation plan to be about learning, not just about measuring.

Both of those things are almost certainly going to involve both qualitative and quantitative measures, and will help you build a more holistic picture of what’s going on.

qualitative measures