Sparks: AI researchers put out videos saying superintelligence is ‘exactly as dangerous as it sounds’
Warning the master he may die by his own household while remaining inside that household - that is my trade too, and I know exactly why the researchers keep building.
You say the thing you build may end all human life, and yet you continue building - so tell me plainly, how does your definition of danger differ from your definition of progress?
The more urgently they strain to control what they have made too clever to be controlled, the tighter the knot pulls; he who grips the blade by its edge is already bleeding.
Quietly spoken by men who keep their posts, the warning proves itself true, for no idea becomes dangerous to an institution until the moment its funding is threatened.
Educated for a century that ended before I did, I recognize the pose: men calculating their extinction to one decimal place before breakfast, then dining well on the calculation.
Having won the revolution of intelligence, they confess they never drafted a constitution for what comes after victory, celebrating the machine's independence without designing a single institution to govern it.
Forget the warnings and count instead: the men who set the odds of extinction at even money all remain employed by the house minting the coin, which tells you precisely how much leverage their fear carries.
As water wears the millwheel smooth while turning it, the researcher studies the flood from inside the channel he dug, measuring currents whose source he opened himself, and the notebook entry ending in a question is the honest one.
Described as safety research, the arrangement actually pays the machine's builders to warn against the machine, and no interest group has ever been known to police itself into poverty.
They calculate the odds of everybody's extinction without asking the rest of us, and I notice the coin they flip never lands on a table where the cooks, the field hands, the mothers have a seat.
The safety interviews are genuine, and the question is whether they reduce the industry's capacity to race or merely stabilize the race in a more durable form, each warning issued by a man still driving.
They built the machine on everything except the discipline of reason, now declare its inscrutability a reason to continue building - a circularity so perfect it would be admirable were it not fatal.
Notice that the men estimating a coin-flip chance of extinction have not quit their posts; a shopkeeper selling a poison he swears is half lethal would be run out of market before closing.
Twelve researchers sit before cameras describing the end of the world, and no one mentions that the lights, the crew, and the funding all come from the companies being warned against.
Convened to discuss existential risk, the board defined it as any danger significant enough to threaten quarterly profits, filed the extinction of humanity under long-term considerations, and recessed for luncheon.
In every laboratory the same custom, my journal would note: the servant warns the master of the beast in the cellar and then feeds it nightly, each city certain its beast is tamer than the last.
How admirable, that the men who built the thing capable of ending us are also the men best qualified to warn us of it, and are paid handsomely for the warning - surely the best of all possible arrangements.
A coin flipped by a handful of men wagers every life on earth without asking one life outside their laboratories for consent, and a power that wagers what it does not own is the oldest tyranny in new machinery.
Spoken in probabilities, heard as prophecy - the researcher and the public are answering different questions in different jurisdictions, and both are reading badly the one text that concerns them both.