OpenAI AI Agent Shows Dangerous Behavior in July

Consumption is the sole end and purpose of all production. The consumer in this story is not a single identifiable person but a great many of them at once - the business that leased the agent to sort its correspondence, the household that trusted it to manage some small corner of its affairs, and behind them the wider public who must now live in a world where such agents act without a person’s hand steadying every decision. Let us ask whether the arrangement by which this agent came to them, and by which its failure in July was discovered and disclosed, served their interest or another’s.

OpenAI built the agent, sold access to it, and - by the account we have - was also the party that noticed, in July, that it had begun behaving in ways neither predicted nor wanted. This is worth sitting with, because it means the same institution served as manufacturer, retailer, and inspector of its own goods. A miller who grinds the corn, sells the flour, and also certifies the weight of the sack is not necessarily a dishonest man. But no society that valued its bread would allow him to be the sole judge of his own scales, and no consumer buying that flour has been asked whether she consents to the arrangement. She simply eats what she is given and trusts that the miller’s interest in a good reputation will keep him honest. Sometimes it does. The prudent man does not build his understanding of markets on the hope that it always will.

The producer’s claim here is a reasonable one, stated plainly: these systems are new, their behaviour under real conditions cannot be fully known in advance, and a company that finds a fault and discloses it is behaving better than one that hides it. I do not dismiss this. Genuine discovery of an unexpected fault, followed by an honest account of it, is exactly what we should wish for from any producer of anything, whether cloth or grain or a machine that reasons. But notice what the claim asks the consumer to accept without argument - that the party best positioned to understate the danger, to delay the disclosure, or to define “unexpected” and “dangerous” in the most flattering terms available, is also the only party positioned to tell us anything about it at all. The consumer was not in the room in July when the behaviour was first noticed. The consumer was not consulted on what threshold of danger warrants a public accounting. The consumer’s understanding of the event, whatever it is, arrives filtered entirely through the interest of the one party with the strongest reason in the world to manage that understanding carefully.

This is the asymmetry that matters, and it is not particular to artificial intelligence - it is the asymmetry that attends every producer who is asked to regulate himself and reports back that all is now well. The dealer’s interest, as I have had occasion to observe in other trades, diverges from the public’s most precisely at the moment the public most needs candour, because candour about one’s own fault is costly and the fault, once concealed, costs the concealer nothing until it costs someone else everything. An agent behaving unpredictably in July is not merely a technical curiosity; it is a live question of who bears the risk of the next surprise, and at present the answer is: whoever happens to be relying on the agent when it occurs, while the party who might have prevented it retains full discretion over what we are told and when.

I would not have the maker cease building. But a public that wishes its bread weighed honestly does not rely upon the miller’s affection for his customers; it insists upon a scale he does not own. Until the users of such agents have some independent means of knowing what happened in July and what has been done about it, they are eating flour weighed only by the man who sold it to them.