Uncontrollable AI threatens global disaster

This policy benefits perhaps a few dozen souls - Timothy Garton Ash, a handful of Silicon Valley experts, a Stanford friend passing warnings over dinner - with the pleasure of foresight, of being early, of feeling the machinery of history turn beneath their own hands. It harms, at the outer bound of the calculation, every human being alive, in the amount of a catastrophe the report itself compares to Hiroshima: sudden, total, irreversible. The arithmetic is uncomfortable, but the arithmetic is the argument. Let us count.

The population at risk is not Silicon Valley. It is everyone - the shopkeeper in Lagos, the schoolteacher in Manchester, the child not yet born who will inherit whatever equilibrium recursive self-improvement leaves behind, should the experts’ timeline of a couple of years prove accurate. Against this near-universal exposure sits a warning system of remarkable narrowness: a journalist, some engineers in one American valley, and a friend from one university. The pleasure of possessing this knowledge is real and ought not be dismissed as vanity - foreknowledge lets a man prepare, invest, relocate his conscience if not his person. But it is a thin pleasure, enjoyed by few, while the pain it forecasts, if realized, would be borne by billions who never heard the warning at all.

Consider the dimensions properly, as I have always insisted one must - not intensity alone but duration, certainty, propinquity, extent, and fecundity. The intensity of a civilization-altering AI failure is not in dispute; even skeptics of the timeline concede the magnitude if the event occurs. The duration is total, since the harm in question is not a bad quarter’s profits but the permanent loss of human control over outcomes that matter. The certainty is contested - this is precisely where the calculus must be honest rather than alarmist - but the propinquity, on the experts’ own account, is now measured in a couple of years, not decades, which changes the moral weight of inaction considerably. A danger forty years off permits deliberation; a danger two years off demands it now, from whoever holds the levers.

Here is where I part from those who would simply amplify the alarm and move on. The strongest version of the case against publicizing this warning is not cowardice but a real mechanism: broadcasting an arms-race dynamic can accelerate the very race it warns against, since firms which believe a rival is close to a breakthrough have every incentive to sprint rather than pause, and public panic can hand advantage to whichever lab is least scrupulous about safety in the scramble for capital and talent. This is not a trivial objection. A legislator who ignores it is not being rigorous, merely loud.

But the calculus still comes out against secrecy, and here is the mechanism that decides it. The warning as it presently travels - from expert to journalist to a friend at a dinner table - produces exactly the wrong distribution of foresight. It reaches people with reputational capital and access to opinion pages, and it fails to reach the regulators, legislators, and publics with actual power to compel disclosure, slow deployment, or fund independent verification of the labs’ own claims about their proximity to recursive self-improvement. Picture the scene plainly: a Stanford friend, at a table in Palo Alto, describing quietly what may be coming, while a legislator in Washington or Westminster reads about it, if at all, as a piece of colour in a broadsheet essay. That is a fire bell rung in the parlour of the wealthy while the exits from the building remain unmarked for everyone else. The fecundity of secrecy - its tendency to produce further secrecy, further private hedging, further advantage for those already positioned to benefit - compounds the original harm. Each dinner-party warning that stays a dinner-party warning is a small pleasure purchased against a large, diffuse pain distributed to people who never got to vote on the risk they were made to bear.

The reform implication follows without much need for rhetoric. If experts genuinely believe recursive self-improvement is a couple of years away, the rational legislator does not wait for the warning to filter down through friendship networks and opinion columns. He mandates disclosure of frontier capability benchmarks to a public body with the technical competence to assess them, independent of the firms racing toward the breakthrough, and he does it now rather than after the first uncontrolled incident makes the argument for him. The objection about accelerating the race is answered not by silence but by coordination - a disclosure regime that binds all major labs equally removes the individual incentive to sprint in the dark, since no one gains an advantage from concealment if concealment is no longer possible.

Every day the warning remains a matter of private conversation among the fortunate few, the aggregate account grows more lopsided: small, concentrated pleasures of insider knowledge on one side, the unweighed, uncounted, and uninformed exposure of everyone else on the other. A calculus that permits this imbalance to persist has not been properly performed. It has simply not been performed at all.