Saturday, April 30, 2005


WYMIWYG - What You Measure Is What You Get

This is another very important system principle. Systems are distorted by the presence of targets, which increasingly fail to measure what they were supposed to measure.

Here is a well-known example. If you test, measure and analyse the ability of school-children, this may tell you all sorts of useful things about the socio-geographic distribution of ability, about the success of different teaching methods, and so on. But if you set targets for the scores achieved on these tests, this will motivate some changes in behaviour among teachers, parents and children. Teachers will "teach to the test", while many middle-class parents will send their children for special coaching. (Indeed, for many schools, the excellent results achieved in the school league tables are in large part due to the huge amounts of extra tuition received by children outside school, and bears little relationship to the amount of added-value provided by the school itself.) While this may indeed improve the scores, it seriously undermines your ability to learn anything useful about the ability of school-children, or to make any systematic changes.

Another interesting example came up in the UK election. Given a reasonable-sounding goal that health clinics should try to see patients within 48 hours, the government had imposed a target. It emerged (on live television, to the surprise and embarassment of the Prime Minister) that in order to reliably achieve this target, some clinics had changed their appointment policy and were now refusing to book appointments more than 48 hours in advance.

Jenni Russell When you can't see a GP (Guardian, April 30th, 2005)

The present Labour government is often criticized for its obsession with targets, but it should be remembered that this obsession was shared by the Conservative government under John Major. The desire to set measurable targets often comes from quite sincere motives, but these targets have dysfunctional effects. This is especially true when targets are set by politicians under political pressure without proper systems analysis.
  • Setting isolated targets for improving the things you are unhappy about, while failing to set targets for maintaining the things you are happy about.
  • Setting sample targets as illustrations of the things you could improve, which then receive disproportionate amounts of attention and resource. ("We will cut waiting times for breast cancer" ... hang on, what about other forms of cancer such as prostate?)
And of course target-setting is not just a disease of politicians. It is also an occupational hazard of managers within organizations. We cannot reasonably demand an end to targets, but we need to work on developing wiser targets.

POSIWID - Purpose of System Is What It Does

At one level, POSIWID shows us how complex systems resist simplistic attempts to change them, or sometimes even to monitor them. The education system contains a testing subsystem whose purpose is to achieve high scores. Tracing how the testing subsystem actually achieves high scores reveals some important dependencies: the test results are dependent on the coaching subsystem, which in turn is dependent on the social system of the parents.

If we want to make meaningful changes in the education system, it is very useful to carry out this kind of dependency analysis, because it helps us to predict several things:
  1. How successful a given initiative will be.
  2. How quickly this success will become visible.
  3. How unequally this success may be distributed in different areas.
  4. How quickly this success may be eroded or undermined by other system effects.
At another level, we can ask about the POSIWID of the target-setting system itself. The avowed purpose of target-setting is usually something to do with accountability or performance or both. The actual effect is often to make things more complicated and more bureaucratic, achieving local accountability or performance only at the expense of accountability and performance elsewhere in the larger system,

It is possible to intervene into this system too, but it needs an intervention at the right logical level. Perhaps incredibly, the quality standard ISO 9000 (dismissed by many people as hopelessly bureaucratic) contains a defined point where you can plug in a control loop that will limit or even reverse the growth of bureaucracy. I really enjoy that kind of intervention - it's like one of those martial arts where you simply redirect the energy of your opponent.

Technorati Tags:

1 comment:

Rob Barratt said...


I'm not convinced that target setting has anything to do with education in its true sense. Targets are driving many teachers round the bend and mean little to pupils. I think they've also become endemic in society generally but no one seems to be challenging their use or their worth. I wrote this poem:

Targets, Bloody Targets by Rob Barratt

Did you meet your targets?
Did they make you SMART?
Or did they mutate into something you hate
A dartboard for your dart

Who decided targets work?
Why play these silly games?
When did objectives become ineffective?
Whatever happened to aims?

Targets are for archery
Or a massive corporation
But when they are set and then are not met
They become an abomination

How did we live before them?
And without them get this far?
How could we believe we could ever achieve,
And who IS the target tsar?

What targets did Da Vinci have
Or Michelangelo?
What targets do you give to an Inuit
Who’s hunting in the snow?

What targets for a drummer
As he plays a paradiddle?
Are violinists’ targets based
On figures they can fiddle?

Targets are wondrous, magical things,
They have a life of their own,
But you have to feel guilt and blood will be spilt
If you’re not in the target zone

But now I’ve hit my target,
‘Cos I’m using my ability
To write a poem based on
This pointless word’s futility

This tiresome word has now become
Redundant psycho-babble
And the only time it’s useful is
When you are playing Scrabble

Next time you’re given targets
Just say that you’re above it,
Fulfil your ambition and don’t feel contrition
By telling them where they can shove it

(sung to the tune of “I don’t believe in if anymore”)
No, I don’t believe in targets any more
They’re an illusion
They’re an illusion

Targets, targets, everywhere,
And all the boards did shrink
Targets, bloody targets
I think that targets stink!!

Yours, Rob Barratt (Bodmin)