AI is eating website traffic, websites are blocking AI - and reliable information is getting harder to find

Every market town lives by the road that runs through it. The traveller stops because stopping is what the road is for; the stall-keeper lays out his goods knowing the pause will pay him. The open web was built on the same quiet arithmetic. The writer set his work where any could take its measure, the searcher paused, and the pause paid for the writing.

Now a machine meets the traveller at the gate, takes his question, hands him the answer, and sends him home. The stall stands empty. The machine’s owners call this the liberation of knowledge from its gatekeepers. I have heard merchants explain their own interest as the public’s too often to accept the phrase unexamined.

The open web is a common pasture. Each machine that grazes it acts sensibly, and the grass dies. The blocking now spreading across it is not malice but enclosure - publishers fencing what the scrapers declined to treat as property. When the channel that made generosity profitable is severed, generosity stops. Blame the design, not the men; a badly built institution produces exactly this as efficiently as a good one produces plenty.

I have spent my life arguing that self-interest, well channelled, serves the public better than benevolence ever manages. This troubles me more than any monopoly I have audited.

And the reader? The workman confined to one operation loses the faculties he never exercises; the reader fed one finished answer loses the habit of weighing. The answer arrives stripped of its provenance, judged by a thing that keeps no spectator in its breast - that cannot blush at an error, and so has no reason to avoid one.

Repair the bargain: make the passage paid and the ledger honest. Otherwise we shall have answers beyond counting, and a famine of the people who know how answers are made.

The web has begun to block the machine that learned from it. Observe the sequence, for it is instructive: first the giving, then the taking, then the lock on the door - installed only after the house stands empty.

The old bargain was never spoken aloud. Content was offered freely so that visitors would come, and the visitors’ arrival paid the makers. The official self-description - everything open, everything free - excluded one question: who, in the end, feeds the feeders? That question did not vanish. It returned as the traffic report, each month thinner than the last.

The machine now answers every question with perfect fluency and cannot say where it learned. This is the remarkable symptom: knowledge without provenance, conviction without a source. A patient who reports the dream but has lost every association to it is not cured; he is worse off than before, for now the dream masquerades as memory. So too the answer that arrives severed from its origin masquerades as knowledge. Reliable information grows scarce not because there is less of it, but because the thread back to its making has been cut - and no one thought to ask what would happen when it was.

And the blocking? One cannot forbid a dream after it has been dreamed. The crawlers already took what they needed. The lock is not a defence but a protest - the system raging against its own earlier generosity, which is the least effective form of rage there is.

Note also the repetition. Newspapers, musicians, encyclopedists - each gave the work away and discovered too late what free had cost. The web watched every one of these funerals and repeated the arrangement anyway, one layer deeper. Systems do this. They repeat what they cannot remember. The web is now the patient on the couch, describing a childhood it insists it never had.

The bargain was never written down, and that is why it could be broken without breaching a single clause. For twenty years the open web ran on an unwritten settlement: the writer published, the index pointed, the visitor arrived, and the visitor’s attention paid the printer’s bill. Traffic was the currency. Trace the money and you had the whole constitution of the thing.

Now the machine answers the question itself. It consumes the writing, keeps the visitor, and remits nothing - or a pittance no publisher can bank on. The payment rail is severed, and no quantity of language about the free flow of knowledge will carry the freight. An information commons is not fed by goodwill, any more than an army is fed by patriotism. It is fed by a revenue arrangement, and the arrangement has failed.

Observe each operator’s rational response. The publisher bolts the door. The machine, starved of fresh material, trains on its own exhaust and whatever slop remains unguarded. The reader receives confident answers assembled from sources that can no longer afford to exist. Every actor behaves sensibly; the aggregate is ruin. I have spent my life building institutions on the assumption that operators will serve themselves, and whoever assembled this one assumed they would not. That assumption is the defect.

It will be said that the machines learn as any reader learns. But a reader who could copy the library instantly and never once buy a book is not a reader; he is the end of books.

The remedy is not exhortation but plumbing: compel the taker to pay the maker - by license, by levy at the point of extraction - not as charity but as specification, aligning the taker’s profit with the maker’s survival. Until then, expect walls, a thinning commons, and a machine answering fluently, at length, from an emptying warehouse.