The Crack You Cannot See
Why AI governance is not really about controlling AI, and why that matters to you
On the morning of 20 March 1905, the workers of the R. B. Grover shoe factory in Brockton, Massachusetts, were arriving for the day shift when the boiler let go. The explosion threw the boiler through the roof. The four-story wooden building collapsed in on itself, and the wreckage caught fire almost immediately. Fifty-eight people died. About 150 more were injured. Many of the dead were trapped in the collapse and scorched where they lay.
The technical cause was almost dull in its predictability. The boiler used a lap seam, two steel sheets overlapping and riveted—a standard, familiar design. Yet this method hid a flaw: a crack could quietly snake along the inner edge of the overlap, invisible to any inspector peering from the outside.
Here is the part worth sitting with. Steam boiler: This is the time to pause and reflect. By 1905, steam boilers were old news, driving mills, ships, and locomotives for decades. Engineers had mastered the science—metallurgy, pressure, fatigue. The equations were all there. What was missing was the world around the boiler: who could inspect it, by what rules, how often, who could order a shutdown, who had to keep records, and what happened to the owner who chose ignorance. Better boiler. It was a set of human arrangements. Massachusetts passed “An Act Relating to the Operation and Inspection of Steam Boilers” in 1907, two years after Brockton, and only after manufacturers had finished objecting to what they called needless government interference. The American Society of Mechanical Engineers convened a committee in 1911. The first Boiler and Pressure Vessel Code appeared in 1914: a single volume, 114 pages. That code, now much larger, still governs pressure vessels today. It exists because ordinary people in a shoe town refused to accept that the explanation was “ the boiler failed.”“”
As a historian, I began with this story for a reason: we are reenacting the same debate, falling into the same category mistake all over again.
The category error
Most writing on AI governance treats AI as a beast to be tamed. Constrain the model. Test it. Red-team it. Certify it. Watermark its output. These efforts matter, and I am glad they exist. But it is like debating lap seams while ignoring the absence of an inspection system.
In reality, an AI system is not simply a model—it is a whole assembly line. It is the model, the data someone selected, the threshold someone set, the workflow someone mapped, the interface someone designed, the target slipped into a quarterly plan, the rushed procurement, the skipped training, the escalation path that may or may not finish with a human, and the culture that either welcomes or silences the words, “this output looks wrong to me.”
A model does not deny anyone’s claim. A workflow denies the claim. A model does not decide that a teacher’s professional judgment is now advisory. A principal’s implementation decision does that. A model does not decide that four minutes per case is enough. A staffing budget decides that.
So when we say “AI governance,” we should mean the thing we can actually govern: the human practices, decisions, and organizational arrangements through which AI is adopted and made consequential. Not the algorithm. The institutional context that turns an algorithm into a consequence in someone’s life.
This shift redraws the circle of who belongs in the conversation. If governance is just about the model, only engineers get a seat, and you are left outside. But if governance is about the arrangements, then suddenly the most important questions are the ones you have been asking all along:
What are we actually trying to achieve here, and how would we know if it worked?
What are we no longer doing because this tool is doing it?
Who gets to say no, and what happens to them afterward?
When it is wrong, who finds out, and how quickly?
Who bears the cost of the errors, and are they in the room?
What did we stop measuring when we started measuring this?
You do not need to know what a transformer is to ask these questions. Every one is a governance question. Yet in most organizations, these questions stay unspoken.
Most AI governance material misses the mark in one of two ways. It is either packed with jargon for engineers—evaluation harnesses, model cards, drift monitoring—or it floats in the clouds, listing principles like fairness, transparency, and accountability that everyone nods at but no one can use. The real power sits with the person in the middle. Choose the tool. You are responsible for what it does. You are also somebody’s parent, and there is a version of this conversation happening at your kitchen table about homework, chatbots, and what your fourteen-year-old is talking about. You hold more influence than you think, but less authority than you wish. That gap is the heart of the problem: those who understand the system best are often the least empowered to change it.
This is not a technology problem. It is an organizational one.
A method: See, Judge, Act
There is a time-tested workplace method for this, older than AI itself. In the 1920s, Joseph Cardijn created it for young Belgian factory workers who lacked power and expertise yet not perception: See, Judge, Act. Look honestly at what is happening. Measure it against what should be happening. Then take one concrete step.
It sounds simple, but it is easy to get wrong. Most people leap from a fuzzy observation straight into frustrated action. To help, I want to borrow a discipline from Eli Goldratt’s Theory of Constraints, designed for this very challenge: how someone inside a system can diagnose it honestly and spark change without formal authority.
SEE: Find the constraint, not the symptom
Goldratt’s first step is to find the constraint. He insists—though people often forget—that a system has very few constraints, usually just one. Improving anything else does nothing. It only shifts the pile.
Start where he starts: list the undesirable effects. Not opinions ~ measurable effects. Cases are being closed faster but reopened more often. Junior staff has stopped asking questions. Nobody can explain last month’s numbers. Two people have quietly built their own workarounds.
Then trace them back. Goldratt’s Current Reality Tree is a formal tool, but you can get most of the value with a pen or pencil: draw arrows from effects to causes, keep asking why until the arrows meet. They almost always meet somewhere unglamorous. Not the model. More likely: nobody owns the override decision, or the metric rewards volume while the risk lands months later on another team, or the purchase happened before anyone defined what good looks like.
Write the constraint down as one sentence.
If you cannot say it, you have not found it yet.
In my experience, the main constraint is rarely accuracy. It is accountability latency—the gap between when a system errs and when a human discovers it. Shorten that gap, and many problems vanish. Leave it long, and no model quality will save you.
JUDGE: Surface the conflict and attack the assumption
This is where most people hesitate, thinking judgment demands technical authority or moral certainty. It needs neither. It only asks for honest naming of the conflict.
Goldratt’s Evaporating Cloud is built for this. The claim behind it is bracing: real, persistent conflicts are not conflicts of will. They are conflicts produced by a faulty assumption that everyone has stopped noticing. The way through is not compromise. It is finding the assumption and breaking it.
The AI version of the cloud shows up in nearly every organization I have seen.
To serve people well, we must move faster, so we must automate the judgment step. To serve people well, we must not harm anyone, so we must keep the judgment step human.
Framed like this, it sounds like an unavoidable trade-off. But it is not. Hidden assumptions lurk beneath: that every case needs human review, not just the crucial ones; that speed and review draw from the same pool; that high-stakes decisions cannot be flagged ahead of time; and that reviewing is slow because humans are slow, not because the interface obscures the machine’s reasoning. And humans being slow is a good thing in decision-making.
Challenge just one of these assumptions, and the conflict can dissolve without negotiation. That is a real, tangible outcome, and you do not need to be an engineer to spark it. You only need to be the person in the meeting who asks, “We are treating this as a trade-off. What are we assuming that makes it one?”
Judging has a second part, and it is not technical: By whose Standard? Fairness, dignity, due process, the right to an explanation, the right to a human. Institutions do not find these in data. They inherit them from law, professional ethics, civic and religious traditions, and from what those affected will accept. Choosing which standards apply is not an engineering job. It is one of the oldest political acts, and right now it is quietly slipping into the hands of procurement.
ACT: Make it small, make it owned, and expect the resistance
Goldratt’s following steps are: exploit (squeeze everything from the constraint before spending), subordinate (align everything else to serve it), elevate (invest only then), and repeat. Do not let inertia become the new constraint. The order matters. Most organizations leap to improve—buying tools or hiring consultants—because that looks like leadership.
(A note here: having been a consultant, a strategist for many global companies, Consultants should never provide answers; they should assist you in the discovery of the correct solution that is best for your organization. And no, I don’t put lawyers and accountants in the same batch as consultants; they are technicians, just like engineers, machinists, and skilled tradespeople. Ok I will explain in a future blog posting.)
Start with exploit. It is almost always free and unglamorous:
Write down who holds decision rights: who can override, who must be informed, who signs off. One page. The act of writing it often reveals that no one actually knew.
Track the reversals. Log every override and complaint, and put those numbers in front of the same people who see the efficiency stats. Right now, one is on the dashboard, and the other is buried in someone’s inbox.
Assign ownership to a person, not a committee. Committees cannot be held accountable, which is often exactly why they are chosen.
Run a pre-mortem. Imagine it is eighteen months from now and the story has gone wrong in the newspaper. What happened? People know. They will tell you if you ask in the third person.
Ask the people on the receiving end. Those affected are the cheapest and most accurate source of insight in the whole system, yet they are rarely asked.
Then expect resistance, and do not take it personally. Goldratt mapped the layers of resistance, and they arrive in order: there is no problem, I disagree with that direction, that will not solve it, yes, but it will cause something worse, there are obstacles you do not see, and finally the silent fear no one names. You cannot bulldoze through these. You must address them one by one. Skip a layer, and you may win the argument but lose the chance for real change.
Outside the building
All of this works just as well at a school board meeting, a council session, or a candidate’s town hall, or in your own household. The method does not change, wherever you bring it. It is even used in classrooms to help students.
See: Ask what systems are already in use, where they are used, and what evidence supports them. In most places, this question has never been asked in public, and the answer is usually either “we don’t know” or a procurement contract no one has read. Judge: Ask whose Standard applies and what occurs when something goes wrong. Act: Ask for something small, practical, and enforceable, like a public register of automated systems in public services, a named accountable official, a real appeals process to a human with authority to reverse decisions, and disclosure when a decision about you was made or influenced by a machine.
Notice that none of these requests are technical. Notice, too, that none of them are guaranteed to you today.
The reason to demand the simple version is that it can be enforced. “Ethical AI” cannot be audited. But “Every automated denial must be reversible by a named human within ten working days.” Or whatever agreed time necessary to see the change and the cause/effect.
Back to Brockton
The 1914 code was 114 pages long. Engineers wrote it, but they were not the reason it happened. It was caused by fifty-eight funerals in a shoe town, by newspapers that kept the story alive, by legislators who were pushed to act, and by manufacturers who lost an argument they thought they would win. The lap seam was the technical cause. The real cause was the lack of an inspection system, and that absence was a choice made by no one in particular. That is often how the most important choices are made.
Right now, we are at the point in the AI story where the needed arrangements do not yet exist, and objections still ring out as “needless interference.” The systems rolling out this year in hospitals, schools, benefits offices, hiring pipelines, and courts will shape lives long before these questions are settled.
You do not need to grasp the technical model. What matters is seeing clearly, judging by a standard you can name, and acting on something small enough to finish. Then repeat, and do not let inertia become the main obstacle.
Someone needed to write the first page.
Sources for the historical account: Grover Shoe Factory disaster (Wikipedia); The Grover Shoe Factory Disaster Shakes the Nation in 1905 (New England Historical Society); The History of ASME’s Boiler and Pressure Vessel Code (ASME); ASME Boiler and Pressure Vessel Code — Engineering Landmark (ASME).