Chapter 56 — What the Tests Found

The four tests ↑

Test One — The Easy Correction

Begin with the easiest possible case.

An error can reasonably be established. There is little genuine disagreement about what happened. Nobody appears to have a substantial personal or institutional interest in preserving the mistake. The institution has the information and authority required to put it right.

Surely correction should be simple.

The test showed that even here, correcting the original error may not be enough.

An error can travel. It may be copied into other records, relied upon in later decisions or produce consequences for the person affected.

The emerging sequence became:

visibility → detection → challenge → acknowledgement → preservation → examination → correction → propagation → remedy → verification → learning

The important finding was:

Correction is not complete merely because the original error has been corrected.

A Corrigible system must also ask where the error travelled, what depended upon it, what consequences it produced and whether those consequences can be repaired.

And when reasonable corrective action has been completed, the matter needs to be capable of closing without becoming permanently immune to material new evidence.

Detailed investigation: Chapter 52 — The Easy Correction →

Test Two — When Reasonable People Disagree

The second test removed certainty.

What happens when competent and reasonable people can examine the available evidence and reach different conclusions?

A system may still have to make a decision.

But making a decision does not make the uncertainty disappear.

The test therefore shifted from asking whether the system reached the right answer to asking whether it accurately preserved the evidentiary character of what it knew.

evidence → competing interpretations → decision under uncertainty → preservation of provenance, dissent and uncertainty

The principal finding was:

A decision made under uncertainty should not acquire greater evidentiary certainty merely because the institution had to make it.

This matters particularly when information travels.

An allegation can become an assessment. An assessment can become a diagnosis, classification or administrative fact. Later people may receive the conclusion without receiving the uncertainty from which it arose.

Repeated references to that conclusion can then appear to corroborate one another even when they descend from the same original source.

So a Corrigible system needs to preserve not only what was decided, but also where the information came from, what was disputed, what remained uncertain and what evidence could justify reconsideration.

Detailed investigation: Chapter 53 — When Reasonable People Disagree →

Test Three — When the Same Thing Keeps Happening

The third test stopped looking at one case.

Suppose a system receives a complaint, investigates it and corrects the problem.

Then something similar happens again.

And again.

At what point do separate incidents become evidence of something larger?

case → correction → recurrence → pattern detection → systemic response → later observation → verification

This produced an important distinction:

Case correction is not necessarily system correction.

A complaints system may become very efficient at repairing individual failures while leaving untouched whatever keeps producing them.

Complaints can therefore perform another function. When examined together, they may reveal relationships that cannot be seen within any individual case.

But pattern detection is not pattern proof.

Low complaint numbers do not necessarily demonstrate good performance either. People may not know how to complain, may abandon the process, may fear consequences or may no longer believe complaining will achieve anything.

So the system has to look beyond the number of complaints and ask whether the underlying conditions actually changed.

The principal finding was:

A system demonstrates learning more convincingly when later evidence shows that its correction changed the conditions that produced the failure — and when contrary evidence can make it change again.

Detailed investigation: Chapter 54 — When the Same Thing Keeps Happening →

Test Four — When Correction Becomes Costly

The fourth test introduced power.

Suppose correction threatens reputation, authority, relationships, previous decisions, money or institutional standing.

The evidence has not necessarily changed.

What has changed is what the institution may lose if the evidence is accepted.

That gives us the Power gradient.

The test asks whether evidence receives the same intellectual treatment when the person or institution being challenged possesses substantially greater power than the person raising the challenge.

Independent review helps, but Independence of the reviewer alone may not be enough.

The institution may still control records, terminology, access to staff, earlier interpretations and the information supplied to the reviewer.

The test therefore asks:

Can the procedure reduce the effects of the Power gradient sufficiently that the evidence — rather than the parties' relative institutional power — determines what can be established?

Then the experiment became harder.

People within an institution may have divided loyalties. One may protect a colleague. Another may protect employment. Another may fear making an unjust accusation. Another may protect reputation. Another may simply trust an earlier decision.

Those people do not necessarily need to coordinate.

Their individual decisions may nevertheless change how corrective information travels through the institution.

challenge → Power gradient → divided loyalties and institutional interests → altered information pathways → scrutiny and correction → remedy

And if earlier responses have themselves produced later decisions, the cost of correction can grow.

Correcting the original problem may eventually require an institution to reconsider not only what happened, but also what it subsequently did about what happened.

The principal finding is therefore broader than the existence of a complaints procedure:

A correction system is tested most severely when correction threatens something the system has an interest in preserving.

Chapter 55 deliberately leaves a larger question open:

What determines whether divided loyalties remain ordinary institutional pressures, converge into resistance to correction, or cross the line into deliberate concealment?

That question requires a larger investigation than Test Four can responsibly answer.

Detailed investigation: Chapter 55 — When Correction Becomes Costly →

What the four tests produce together ↑

The tests progressively removed the protections that made correction easy.

First we removed the assumption that correcting one record was enough.

Then we removed certainty.

Then we removed isolation and allowed failures to recur.

Finally, we removed institutional neutrality and introduced power and consequence.

What remained was a model of corrigibility that can be expressed quite simply:

Can the problem be seen?
↓
Can it be challenged?
↓
Can the evidence be examined without manufacturing certainty?
↓
Can the correction follow the error wherever it travelled?
↓
Can repeated failures be recognised as a pattern?
↓
Can correction survive differences in power and institutional interest?
↓
Can the consequences for the person be repaired?
↓
Can later evidence show whether the correction worked?
↓
Can material new evidence make the system reconsider its answer?

That final question matters.

A system is not Corrigible merely because it possesses a correction procedure.

It must remain capable of discovering that its correction was wrong too.

The resulting model ↑

The four tests suggest that a Corrigible system needs several capacities working together.

It needs visibility, so errors can become known.

It needs challenge, so people affected by authority can question authority.

It needs Provenance, so conclusions remain connected to the evidence from which they came.

It needs Preserved uncertainty, so a necessary decision does not become artificial certainty.

It needs Correction propagation, so correction follows an error into the places affected by it.

It needs the capacity to recognise Recurrence, so repeated cases can reveal possible systemic problems without treating similarity alone as proof.

It needs sufficient Independence when institutional interests or Power gradients threaten the integrity of review.

It needs remedy, while keeping Remedy burden visible, because improving the system for the next person does not repair the person already harmed.

It needs verification, because implementing a recommendation does not establish that the recommendation worked.

And it needs Learning persistence, so what was learned survives changes in personnel, policy and time.

None of those capacities alone is enough.

Together they provide something closer to a system capable not merely of correcting mistakes, but of learning from being wrong.

A model must remain Corrigible too ↑

These tests have produced a model.

They have not produced a final answer.

That distinction is important.

The model should now be exposed to other institutions, other kinds of decisions, contrary evidence and circumstances we have not considered.

If those tests reveal weaknesses in it, the model should change.

Otherwise we would finish an investigation into corrigibility by making its conclusions incorrigible.

The test must remain Corrigible too.

That is also why the detailed chapters remain part of the model. They preserve how the conclusions were reached, where the evidence changed the test, and where questions remain open.