Executive Summary
Governance structures are designed to prevent specific categories of organizational failure. Through a systematic mechanism, they reliably produce the conditions for the failures they were designed to prevent. This is not irony—it is physics.
The mechanism: protective structures create the appearance of safety, which reduces vigilance. Reduced vigilance allows drift. Drift accumulates until catastrophic failure—of exactly the type the protective structure was designed to prevent.
The Protection Paradox Mechanism
Consider a risk committee designed to prevent risk accumulation. Its existence signals to the organization that risk is being monitored. This signal reduces individual vigilance—the committee is watching, so individuals don't need to watch as carefully. The committee, meanwhile, develops its own metrics and thresholds, and becomes skilled at identifying the categories of risk it was originally designed to catch. It becomes less capable of identifying novel risks that don't fit its framework.
The result: the organization accumulates risk in exactly the categories the risk committee was not designed to catch, and the committee's existence has reduced the organization's capacity to notice.
Case Studies
Financial crisis audit functions: Bank audit committees designed to prevent the accumulation of dangerous mortgage-backed securities were staffed by people who understood the risks the previous crisis had identified. They were not staffed by people who could identify the new risks being accumulated.
Healthcare compliance programs: Hospital compliance programs designed to prevent billing fraud reduce billing fraud. They simultaneously create the organizational machinery that can be captured for systematic fraud at a different level—the one the compliance framework wasn't designed to watch.
Software security audits: Regular security audits create confidence that security is being maintained. That confidence reduces the informal vigilance that catches the categories of vulnerability the audit framework doesn't check for.
Key References
Normal Accidents: Living with High-Risk Technologies. Basic Books.
The Audit Society: Rituals of Verification. Oxford University Press.
The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. University of Chicago Press.