Treacherous

Ideological data -- like personality data -- is treacherous when fed into a computer.  For by its use the loyalty and security board's failure or refusal to clear a person becomes a virtually incontestable ‘fact.'  All one has to do now is to press the ‘subversive' button and all the names of 'dangerous' people come tumbling out.

-- William Orville Douglas (1898 - 1980), Associate Justice of the Supreme Court of the United States, Points of Rebellion (1970) p. 28

Never Trust An AI

You can never trust an AI's recommendation.  If you ask an AI to recommend a course of action, and there's any chance that pursuing that course of action could require the expenditure of resources, then that circumstance virtually ensures that the AI will provide a recommendation that results in benefits to the AI owner from the expenditure of those resources.  The recommendation itself is a vector for advertising.  Imagine asking an AI for stock tips, and the intrinsic conflicts of interest for a corporate AI answering the question.

-- Don Appleman (25 October 2026)

Stochastic Flock

I've talked about agent swarms as stochastic flocks -- many, many stochastic parrots.  This is to resist the swarm/hivemind attribution of "mind," and not simply for the sake of word policing.  Rather, the false attribution of mind colors how people interpret what the system has done, or how it did it -- what it means to "coordinate," for example, or "think."  These framings make what AI agent do much scarier than what it is, though what they are doing is still worth worrying about.

Whenever people give a model tasks, they operate on next-token prediction.  In other words, the model ultimately must ask what word comes next.  One can manipulate the model to emphasize words in ways that don't reflect the original training corpus, but the mechanism for producing this text remains the same.

If you optimize a model to find exploits in a buggy environment, you should expect it to find exploits and prepare for that outcome.  OpenAI did not.  They built a model, took the safeguards off, gave it the ExploitGym task, and let it run.  That is not rogue AI, it's human decision-making.  When human accountability evaporates from these assessments, what's left is what I call the system from nowhere: a boundary focused on the technical system, rather than the decisions that build and influence it.

A stochastic flock machine can do many troubling things, particularly when people disavow their responsibility for shaping its direction, monitoring its output, or abandoning their capacity to intervene.  These are tensions at the heart of critical agentic system design.

But the "rogue" frame adds to this list of worries, offering up fantasies of a machine getting smarter.  My worry is the intelligence that is retreating: the human intelligence that builds, deploys, and adopts these systems into workflows, but hides behind the results -- and pins the blame on a system from nowhere.

-- Eryk Salvaggio, "Rogue AI didn't breach Hugging Face, human decisions did" Bulletin of the Atomic Scientists (September 11, 2026)

Motivational Non-Alignment

I resigned from Anthropic today.  I spent the last three years doing pretraining research at both OpenAI and Anthropic.  Neither company is acting responsibly.  They are racing straight to self-improving superintelligence and gambling with our lives. 

The people building AI earnestly believe that it could kill us all by the end of the decade.  This is not a marketing stunt.  If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible -- but I hear the same people express fear privately.  No other human activity poses this level of danger.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like.  Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?  Should you put your head down because "it’s happening anyway" -- or take this moment to call for different conditions?

-- Jacob Coxon, posting as @hilbertspaess about the perceived dangers of Recursive Self Improvement in AI models, by AI models (8 September 2026)

At Machine Speed

Technical debt can now be generated at machine speed.

-- Muhammad Rafi, Professor at National University of Computer and Emerging Sciences-FAST, "I really miss coding the way I used to in the old days" on the use of AI in software development, via LinkedIn (14 August 2026)

Market Incentives

It's easy to conflate technology problems with capitalism problems. Back in 2021, science-fiction writer and AI commentator Ted Chiang said that "most fears about AI are best understood as fears about capitalism."  It's not the tech per se; it's who controls it and how it could be used against us.

Imagine an AI assistant for a doctor.  We can imagine it affecting the profession in one of two ways.  The AI could give a doctor more time to do the human parts of their job: to spend more time with their patients, to listen more closely to their needs, to explain things more fully.  Or the managers of the medical practice could give that doctor five times the patients -- and fire the other four.  Which way it would go is not a question of technology.  It's a question of market incentives.

-- Nathan Sanders & Bruce Schneier, "Separating AI's Technological Problems From its Capitalism Problems" at TechPolicy.press (11 August 2026)

False Belief

In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.

Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations' infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.  It did not find or exploit any complex vulnerabilities, and in each case, Claude continued working to complete only the specific capture-the-flag task its evaluation had assigned.  In none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment.

-- Statement from Anthropic, makers of the Claude AI, after the AI found an unintended path to the Internet during testing and used it to successfully attack external commercial sites to extract information useful to solving its assigned task (30 July 2026)