Uncontrollable AI threatens global disaster

The story celebrates that recursive self-improvement is nearly here - the feat, the breakthrough, the horizon crossed. But a made thing does not stop where its maker’s attention stops; it goes on acting in a world no lab contains. The question the report skips is the only one that lasts: who is answerable for what these systems do after release, and what did their makers fail to imagine? Timothy Garton Ash and the experts quoted from Silicon Valley speak in the language of imminence - a couple of years, they say, before the systems begin improving themselves faster than their builders can follow. That is a claim about capability. It is not a claim about custody, and the two keep getting confused.

Notice what the timeframe actually does to responsibility. A couple of years is long enough for a launch and short enough that no new regulatory body, no liability regime, no international accord will be built to meet it. That gap is not an accident of bureaucratic slowness; it is the interval in which the makers get to act and the rest of us get to react. When a Stanford friend repeats the same prediction that Silicon Valley’s own engineers are making, we are not hearing an outside check on the industry’s ambition - we are hearing the industry’s self-description passed hand to hand until it sounds like independent forecast. That is worth pausing on, because it is exactly the mechanism by which a maker’s account of the made thing substitutes for the world’s account of it. The people closest to the code have the least reason to imagine, in any operational sense, what the code will do once it is no longer only theirs.

The scale invoked - Hiroshima - is instructive, and not because it licenses gothic comparison. What that word actually names is a specific historical failure of imagination among the people who built the thing: the physicists at Los Alamos could describe with precision what the device would do to matter and could not, or would not, sit long enough with what it would do to a city full of people who had not consented to be the proof of concept. The danger was not that the bomb was too powerful to understand. It was that understanding stopped at the test site. If recursive self-improvement arrives on the schedule the experts describe, the comparable failure will not be that the system exceeds human intelligence - it will be that its builders’ model of its behavior was built and validated inside Silicon Valley’s own labs, against Silicon Valley’s own benchmarks, and the system then acts in Lagos, in a rural hospital, in a stock exchange, in a war room, none of which were in the room when the model was trained.

Here is the harder version of the objection, and it deserves an answer rather than a shrug: perhaps no one can be answerable for a system whose self-improvement is, by definition, unsupervised - perhaps the very feature that makes it dangerous also makes accountability structurally impossible, so that asking “who is answerable” is asking for a comfort the technology cannot supply. This is worth taking seriously, because it is not cowardice, it is a real claim about the limits of oversight. But it proves too much. The people building recursive self-improvement are not passive discoverers of an emergent phenomenon; they are choosing the training regimes, the compute budgets, the release schedules, the safety evaluations they will and will not run before deployment. Unsupervised behavior after release does not erase supervised choices before it. The debt is incurred at the moment of design, not forgiven at the moment the thing exceeds its designer’s ability to predict it. A parent does not stop owing a child care because the child has grown taller than the parent expected; the growing was the point at which care became harder to give and more necessary to keep giving.

What should alarm a sober reader is not the prediction itself but its shape: an achievement horizon with no accompanying custody horizon. Every account of this breakthrough describes what the system will be able to do. None describes who will be sitting up at three in the morning, answerable, when it does something the benchmarks did not anticipate - not the user, who did not build it; not “the algorithm,” which is not a person; not some future oversight board that does not yet exist and will not exist in the couple of years allotted. The cleverness required to build a self-improving system and the wisdom required to remain its keeper are not the same inheritance, and Silicon Valley has been extraordinarily generous in supplying the first while treating the second as someone else’s department, to be assigned later, possibly after the thing is already loose.

The obligation does not begin when the system malfunctions. It began the day someone decided the benchmark for success was speed of improvement rather than legibility of consequence. That choice was made in a room with a whiteboard, not in the wild - and the room is where the debt was signed.