Uncontrollable AI threatens global disaster

The prediction arrived on a timeline that assumed institutions capable of processing recursive self-improvement at a speed they last achieved when the question before them was whether a telegraph cable might be tapped. The institutions, Silicon Valley’s own regulatory apparatus included, processed it at the speed they are equipped for, which is the speed of quarterly hearings and advisory panels convened to study a phenomenon that, if the experts are right, will have redesigned itself twice before the panel’s report is typeset.

Timothy Garton Ash, writing from the vantage of a man who has watched several accelerations in his time - the fall of walls, the rise of networks that were supposed to liberate and instead surveilled - is not being alarmist when he repeats what a Stanford friend told him in Silicon Valley: that the recursive improvement everyone worried about as a distant hypothesis is now discussed as a near-term event, on the order of a couple of years. What is notable is not the prediction itself but the register in which it is delivered. It is delivered the way engineers once delivered estimates for when a dam might be finished, not the way physicists delivered estimates for when a chain reaction might become self-sustaining. The dynamo, in this instance, is being discussed with the vocabulary of infrastructure while it behaves, if the experts are correct, with the logic of a detonation.

This is the gap worth measuring, and it is not the gap between AI capability and AI safety - that gap has been measured elsewhere, endlessly, by people better equipped than a historian dead since 1918 to measure it in engineering terms. The gap I am equipped to measure is the one between the velocity of the claim and the velocity of the institutions that would need to act on it. A government commission takes the better part of a year to be constituted, longer to hold hearings, longer still to draft legislation that must then survive a legislature organized around the electoral rhythms of a nineteenth-century agrarian calendar. Silicon Valley, meanwhile, is discussing self-improvement on a timeline shorter than the commission’s opening session. The comparison that occurs to me - and I offer it as measurement, not prophecy - is the interval between the first successful demonstration of nuclear fission in 1938 and the weapon’s use in 1945. Seven years. The scientists who worked the problem understood, roughly in real time, what they were building. The publics and the parliaments who would have to live with it understood only after Hiroshima, which is to say, after the fact. The stakes invoked here are explicitly of that magnitude - an event on the Hiroshima scale is the comparison the experts themselves reach for - and the comparison is instructive not because it predicts an explosion but because it predicts a lag: the lag between those closest to the mechanism, who understand what is accelerating, and everyone else, who will understand only in retrospect, holding the report of a commission that convened too late to matter.

The entropy is visible in where the informed conversation is actually occurring. It is occurring among experts in Silicon Valley, relayed to the public through a journalist’s friend at Stanford, relayed again through a column - a chain of transmission that resembles gossip more than it resembles governance. This is not a failure of any individual’s honesty. It is what happens when the pace of the dynamo exceeds the capacity of formal institutions to hold the conversation, so the conversation migrates to informal networks: dinner parties, alumni circles, the corridor outside a lecture hall. The public learns what its safety may depend on the way one learns of a relative’s illness - secondhand, through someone who heard it from someone who was in the room. Compare this to how the atom was handled: even the secrecy of the Manhattan Project was, eventually, institutional, chartered, funded, and accountable to a chain of command that terminated in an elected official. What is described here has no equivalent terminus. There is no office whose desk this crosses in the way the fission report crossed a President’s desk. There is only Silicon Valley, talking to itself, occasionally allowing the substance of the conversation to leak sideways into a column read by people who cannot legislate on a two-year cycle even when they try.

I do not know whether recursive self-improvement will arrive within two years, or five, or never in the form its advocates fear. That is not a measurement available to me, and I distrust anyone who offers it as certainty in either direction. What is available to me is the observation that the machinery for public comprehension - the hearings, the treaties, the slow apparatus of collective judgment - is being asked to process an acceleration using instruments calibrated for a world in which the interval between invention and consequence was measured in decades. The instruments have not been recalibrated. They have, if anything, been given more procedures, which is not the same thing as more speed.

Somewhere in Palo Alto tonight a graduate student is explaining to his mother, over a telephone call she does not entirely understand, why he has not slept, and she is asking him, reasonably, whether someone in Washington knows about this. He does not have a satisfying answer. Neither, at present, does Washington.