On: More US lawmakers seek new AI rules after researchers warn of human extinction

Twelfth of the month, and the American legislature has discovered eschatology.

Two men who make the machines announce that the machines may end mankind, and the men in Congress respond at once with drafts of law. I note the sequence, for the sequence is the whole story. The warning comes from within the workshop. The alarm is sounded by those with the deepest interest in being indispensable. And the legislators, who understand nothing of the mechanism, accept the testimony of its makers as though disinterested scholarship.

The parallel is not new. The medieval popes declared the danger of usury while collecting tithes; the East India Company’s directors reported on the perils of monopoly from the midst of one. When the fox is invited to describe the henhouse, he does not describe it falsely - he describes it accurately, and then asks for larger walls under his own keeping. The researchers who cry extinction are sincere, I do not doubt it. Sincerity is no safeguard. The most dangerous counsel in history has been given by men who believed every word.

And observe what the proposed remedy is: rules, framed by the same governments that cannot audit what they would regulate, enforced upon artefacts no living official comprehends. Accountability requires that someone answer for the thing. Who answers here? The companies answer to their shareholders. The legislators answer to the next election. The researchers answer to no one at all, and it is their word that moves the state.

I have watched empires justify themselves by the chaos they alone could prevent. The device is old: manufacture the terror, then sell the shelter.

The machine may indeed prove mortal to us. But the greater and nearer danger is a power that writes its own certificate of necessity and calls the writing regulation.

Fear is the oldest licence ever granted.