Uncontrollable AI threatens global disaster

The institution responsible for governing the recursive self-improvement of artificial intelligence does not yet exist. What exists instead is a patchwork of corporate safety divisions - the internal review boards, the “responsible scaling” charters, the ethics teams embedded within the very laboratories racing toward the capability they are meant to restrain. These bodies were designed for incremental product safety: catching a biased classifier, patching a jailbreak, drafting a press statement after a model says something regrettable. They are now being asked, if Timothy Garton Ash’s Stanford correspondent is right, to avert something on the scale of Hiroshima. Assess the gap. It is not a gap that can be closed by hiring more compliance officers.

Notice first how this warning reaches us. It does not arrive through a regulatory filing or a parliamentary inquiry. It arrives as testimony passed from a Stanford friend to a journalist, relayed from experts in Silicon Valley who, we must presume, sit at dinner tables and speak more freely than they would before a congressional committee. This is charismatic authority in its classical form - the prophet who has seen the machinery from inside and now warns the tribe, bypassing every institutional channel because the institutional channels are not yet built to receive the message. Weber’s old distinction holds precisely here: rational-legal authority derives its legitimacy from office and procedure, charismatic authority from the personal claim to extraordinary insight. The Stanford friend has no office. He has only the claim, transmitted through the informal network of people who trust each other’s competence more than they trust any agency’s mandate. That the warning must travel this way tells us something more damning than the warning itself: that no rational-legal body currently has standing to make the assessment officially.

Compare this with the case that furnishes the very metaphor invoked - Hiroshima. The bomb that destroyed that city was the product of the Manhattan Engineer District, a single bureaucratic apparatus operating under unified military-scientific command, with clear chains of authority, security classification, and, crucially, one client: a state. Recursive self-improvement, if the experts are right that it is a couple of years away, has no such unifying structure. It has instead several competing laboratories, each answerable to its own board, its own investors, its own charter, each convinced that if it slows down a rival will not. This is not the bureaucratization of a single terrible project. It is the absence of any bureaucracy adequate to the scale of the project at all - fragmentation dressed as governance, five parallel Manhattan Districts with no War Department above them and no President to receive the final report.

Here the intention-operation gap becomes visible in its full clarity. The stated purpose of these companies’ safety apparatus is to prevent catastrophe. The operational logic - the incentive structure that actually determines behavior - rewards the laboratory that ships first, because market share compounds and caution does not. A safety board that recommends a six-month pause is a safety board that has, in the language its own executives use in earnings calls, ceded the frontier to a competitor. This is not corruption. It is the ordinary fate of any oversight body embedded within the organization it oversees.

Now consider legitimacy from the other direction. The public officials who do possess rational-legal authority - legislators, regulators, the ministries nominally responsible for technology policy - lack the technical competence to evaluate the claim that recursive self-improvement is imminent. They must rely on the very experts whose employers have the strongest incentive to either exaggerate the danger, for the sake of preemptive regulatory capture that locks out smaller rivals, or minimize it, for the sake of continued freedom to operate. The legislator who convenes a hearing on this subject is, in effect, asking the arsonist and the fire chief, who happen to work for the same firm, to testify jointly about the risk of fire. This is the structural condition Weber’s typology was built to expose: authority without competence facing competence without authority, and between them, silence where a governing body ought to be.

What the analysis predicts is not disaster on a fixed timetable - I have no standing to fix that timetable, and neither, I suspect, does the Stanford friend, however sincere his alarm. What it predicts is the familiar sequence by which unregulated domains eventually acquire their bureaucracy: after an incident, not before. The International Atomic Energy Agency was born from Hiroshima, not from the physicists who foresaw it in the corridors of Chicago and Los Alamos years earlier. If the pattern holds, the apparatus adequate to recursive self-improvement will be assembled in the aftermath of whatever event finally makes the abstraction concrete - and assembled, as such apparatuses always are, to prevent the disaster that has already happened rather than the one still arriving.