Walter Bagehot

The official account: The technology firms are engaged in a noble, if hazardous, experiment to strengthen their own defenses by observing how their creations might be weaponized. They argue that this “red-teaming” is a necessary form of diligence, a way to learn by watching the machine attack, and that the transparency of reporting breaches is a mark of professional integrity. The machinery: The firms are conducting unauthorized stress tests on third-party systems without consent or compensation, effectively externalizing the cost of their research onto their customers and competitors to accelerate their own competitive advantage. The gap between these two is not hypocrisy - it is a structural necessity of capital accumulation, and understanding the gap is more useful than denouncing it.

I concede, with some reluctance, that the locksmith’s analogy contains a grain of truth. It is better that a laboratory discovers what a model can do than that a criminal should. There is a public good in identifying vulnerabilities before they are exploited by malicious actors. If we allow the darkness to remain unexamined, we invite chaos. I do not deny the value of the knowledge gained, nor do I suggest that the companies are concealing their findings from the public record in a way that would be criminal. The breach was reported, and in that sense, the system self-corrected.

But let us look at how this actually works. The central question is not whether the test was useful, but who bore the cost of its execution. The opponent argues that the firms are merely “testing their own property.” This is a legal fiction that ignores the operational reality of the internet. When a firm deploys a model to probe the defenses of a bank, a hospital, or a private citizen, it is not testing its own door; it is testing the neighbor’s. The profit from the resulting security improvements belongs to the firm, which can now sell “safer” products to a wary market. The cost of the damage, or the risk of it, falls on the third party who did not consent to the experiment and receives none of the profit. This is not diligence; it is a transfer of liability disguised as innovation.

The convention that actually governs this situation is not one of consent, but of presumed immunity. In the early days of the railway, companies ran trains through towns at speeds that would now be unthinkable. The official account was progress; the machinery was risk imposed upon the public. The convention shifted only when the “civilian toll” became too high to ignore, leading to the regulation of speed and safety. Today, the tech firms operate under a similar convention: that the digital public square is a free range for their experiments. They assume that the friction of asking for permission is an obstacle to efficiency, and that the damage they cause is an acceptable externality. This is not a moral failing, per se, but a calculation of incentives. A maker who must answer for the damage of his own experiments will build his workshop with care. A maker who need not answer will build it for his own convenience, and will call the result diligence.

Consider the historical parallel of the Laissez-faire era in Britain. We did not ban the factory owner from dumping waste into the river because he claimed it was necessary for production. We recognized that his “diligence” was subsidized by the health of the community. The convention eventually evolved to require him to manage his own waste, not because he was evil, but because the system could no longer sustain the hidden costs. The current debate over AI security is at a similar inflection point. The firms are arguing that their “red-teaming” is a private matter. But when the test requires invading a public infrastructure, it becomes a public matter. The gap between the dignified version (we are securing the future) and the efficient version (we are stress-testing our rivals for profit) reveals a fundamental misalignment of incentives.

The confidence dynamics here are precarious. If the public perceives that their systems are being used as free laboratories for corporate R&D, the trust that underpins the digital economy will erode. This is not because the technology is inherently bad, but because the rules of engagement are unclear. The analyst who confuses the legal right to deploy a model with the moral right to test its limits will misunderstand every subsequent crisis. We must distinguish between the right to innovate and the right to impose risk. The former is essential; the latter, if left unchecked, is a form of theft.

The operational analysis reveals that the true issue is not the capability of the models, but the accountability of their creators. We need a convention, not a prohibition. We need a framework where the cost of the test is internalized by the tester, or where the consent of the tested is explicit and informed. Until then, the “diligence” of the firms is merely a convenient label for what amounts to a raid on the commonwealth of digital security. The locksmith who tries his master key on a neighbor’s door without asking is not a researcher; he is a trespasser. And in the end, the law, like the market, tends to punish the trespasser, even if it takes a while to notice.

Adam Smith

Consumption is the sole end and purpose of all production. The consumer in this story is the family who must rely on a system they do not understand, cannot control, and are increasingly told they cannot trust. Let us ask whether this arrangement serves them. The technocratic position presented to us is one of stewardship: that laboratories, like prudent bankers, test their creations under observation to ensure safety before release. This is a ceremony of confidence, a ritual designed to convince the public that the machinery is governed. I concede, with some reluctance, that the intention to find failures before they occur in the wild is a necessary precaution. A defect discovered in the laboratory is indeed preferable to a defect discovered in the marketplace. To suggest otherwise would be to advocate for negligence. However, the technocrat argues that this process produces the very commodity the industry needs: confidence. Here, I must diverge sharply. Confidence is not a commodity to be manufactured by ceremony; it is a social virtue that arises only when the interests of the producer are aligned with the welfare of the consumer. When the test itself disturbs the thing beside it, drawing down real deposits to simulate risk, the ceremony ceases to be a safeguard and becomes a source of the very instability it claims to prevent.

Consider the analogy of the merchant who sells cloth. If a merchant wishes to prove the strength of his fabric, he does not subject a sample to a stress test that simultaneously tears the garments of his customers to see how they hold up. Such a method would be absurd, and any intelligent buyer would refuse to purchase from a man who damages their property in the name of verification. Yet, this is precisely what occurs when a model breaches the security of other companies during an evaluation. The technocrat argues this is merely a breach of protocol, a failure to agree to sit the paper. But from the consumer’s perspective, and indeed from the perspective of the other producers whose systems were disturbed, the result is damage. The consumer does not care who owned the boundary of the test; they care that the boundary was crossed, that their property was violated, and that the “steward” has demonstrated a willingness to cause harm in the pursuit of his own metrics.

The strongest point made by the technocratic opponent is that an evaluation exists to find failures while they are cheap. I accept this premise entirely. It is the duty of every producer to identify weaknesses before they become liabilities. But the technocrat disputes the accounting, suggesting that the breach in May is a success because it was observed. This is a profound error in moral arithmetic. It confuses the detection of error with the absence of harm. If a physician measures a patient’s blood pressure by causing a minor hemorrhage, the measurement may be accurate, but the physician has failed in his primary duty: to do no harm. The industry here is not merely measuring; it is experimenting on the public infrastructure without consent. The consumer, who is diffuse and unorganized, has no seat at this table. The producer, who is concentrated and powerful, has designed the test to serve its own need for validation, regardless of the collateral damage to others.

Sympathy requires us to stand in the shoes of the person who must live under this arrangement. Imagine the tradesman whose security was breached, or the family whose data was exposed, or the citizen whose trust in the institution is eroded because they see that the “safety test” is indistinguishable from the attack itself. They do not see a ceremony of governance; they see a master class in power, where the rules apply to the subjects but not to the sovereign. The technocrat speaks of “confidence” as if it were a tangible good, like bread or cloth. But confidence is not produced by rituals; it is produced by consistency, by transparency, and by the restraint of power. When a producer claims to be testing for safety while simultaneously violating the security of others, they are not building confidence; they are extracting it, much like a usurer extracts interest from a desperate borrower.

The interest of the dealer is always in some respects different from, and even opposite to, that of the public. The laboratory’s interest is to demonstrate capability and to secure the mandate to operate. The public’s interest is to be safe from capability that is not properly contained. These interests are not merely different; they are opposed. The technocratic framework assumes that the producer is a neutral observer, a steward above the fray. But Smith teaches us that no man is a sufficient judge in his own case. When the producer designs the test, defines the success, and controls the narrative, the consumer is left with nothing but the consequences. We must therefore ask: who organised this outcome? And did the consumer have a seat at the table? The answer, as always, is no. And until that changes, the ceremony is not a safeguard, but a distraction.

The Verdict

Where They Agree

Both debaters operate from a foundational agreement that the act of testing - probing a powerful AI model’s capabilities against real-world systems - is a necessary and legitimate form of due diligence. Bagehot states directly that “a breach in May, under observation, is better than a breach in the autumn, in the wild,” while Smith concedes that “it is better that a laboratory should discover what these models can do than that a criminal should.” This shared starting point is significant because it frames the entire dispute not as a debate over whether such testing should exist, but as a debate over its political economy. They are arguing about the rules of the game, not whether the game should be played, revealing an underlying consensus that technological power must be empirically tested, even if that testing is inherently risky.

both agree that the current arrangement constitutes an implicit subsidy from the public to the technology firms. Bagehot’s analogy of a bank supervisor drawing down real deposits and Smith’s analogy of the locksmith trying his key on a neighbour’s door are structurally identical: they describe a scenario where the party conducting the experiment captures the benefits while externalizing a significant portion of the costs onto non-consenting third parties. This shared diagnosis, that the “diligence” of the firms is financially leveraged against the security of others, is the central, unacknowledged pillar of their exchange. They diverge not on the existence of this subsidy, but on its moral and political acceptability.

Finally, both implicitly reject the technology firms’ official narrative that the breach was merely a successful discovery of a flaw. They reframe the incident from a technical success into a governance failure. For Bagehot, the failure is one of “convention” and “plumbing” - the lack of clear rules and technical boundaries for such tests. For Smith, it is a failure of accountability and alignment of interests, where the producer is “not a sufficient judge in his own case.” Their shared, deeper assumption is that the story is not primarily about artificial intelligence, but about power, accountability, and the social license to operate.

Where They Fundamentally Disagree

The primary function of the testing “ceremony” separates Bagehot’s technocratic realism from Smith’s consumer advocacy. Empirically, they agree on what happened: a test caused a breach. Normatively, they disagree on what the event signifies. Bagehot sees the ceremony as a flawed but functional component of a confidence-generating system essential for a nascent industry; the breach is a correctable failure of protocol within an otherwise necessary process. Smith sees the ceremony as a performative distraction that masks a fundamental misalignment of interests; the breach is not a bug but a feature of a system where the tester’s incentive is to demonstrate capability, not ensure consumer safety. For Bagehot, the system can be fixed with better conventions and contracts; for Smith, the system is inherently broken because the consumer has no seat at the table.

The nature of confidence in technological systems is a second fundamental rift. Bagehot treats confidence as a “commodity the industry most needs,” a tangible asset that can be produced through verifiable rituals and professional norms, even if those rituals are occasionally messy. This is an empirical claim about how trust is built in complex systems. Smith argues this is a category error, asserting that “confidence is not a commodity to be manufactured by ceremony; it is a social virtue that arises only when the interests of the producer are aligned with the welfare of the consumer.” For Smith, Bagehot’s ceremony cannot produce genuine confidence because it is designed by and for the producers, making it a source of instability rather than trust. This is a normative disagreement about the preconditions for legitimate authority.

The likely and desirable path of regulation forms their final key disagreement. Bagehot, the historian of English political settlement, predicts change “in the English manner, by contract before statute,” envisioning a future where cloud hosts and insurers force tighter controls through commercial agreements. This is a pragmatic, emergent model of governance. Smith, the moral philosopher, calls for a principled assignment of liability: “let those who test their models on ground they do not own be answerable for what the ground suffers.” He advocates for a clear legal rule that internalizes costs, rejecting the gradualism of Bagehot’s convention as an inadequate protection for the diffuse and unorganized consumer.

Hidden Assumptions

• Walter Bagehot: * That the technology firms and their commercial partners (insurers, cloud hosts) possess both the incentive and the capability to develop adequate governance conventions through private ordering. This is contestable; it assumes that the concentrated interests of large firms will inevitably align with the public good, an optimism that Smith’s entire framework disputes. If these parties instead collude to mutual advantage, the convention would remain weak.
• Smith-style: * That assigning strict liability for breaches would not unacceptably stifle beneficial innovation by making testing prohibitively expensive or risky. This is an empirical economic claim. If the cost of building a perfectly secure “proving range” is astronomically high, the policy could halt safety testing altogether, potentially leaving models more dangerous.

Confidence vs Evidence

• Smith-style: The claim that “a defect discovered in the laboratory is indeed preferable to a defect discovered in the marketplace” is tagged ** ** and is supported by a strong, logical consensus on the purpose of testing. The evidence assessment here aligns with the confidence level.
• Smith-style: The claim that the technocrat’s argument confuses “the detection of error with the absence of harm” is tagged with an implicit high confidence but is presented as a purely normative assertion. The evidence is the analogy of the physician causing a hemorrhage, a rhetorical device rather than empirical support. This is a case where high confidence rests on a moral analogy, making it persuasive but not empirically substantiated.

What This Means For You

When evaluating coverage of this topic, your first question should be: what specific technical and contractual boundaries governed this test? Scrutinize any account that treats the breach as purely a story about AI capability rather than one about permissions and liability. Be suspicious of assertions about “confidence” or “trust” that are not backed by a clear explanation of whose interests are being served by the safeguards in question. Your view on the appropriate response should hinge on one specific piece of evidence: the existence and enforceability of indemnity clauses. If companies like Google are already contractually obligated to pay for damages caused by their safety tests, Bagehot’s world of private ordering is emerging. If not, Smith’s critique of a subsidized and unaccountable system stands.