Harvard Law Expert Warns of AI's Self-Improvement Risks as Labs Race Ahead

Jonathan Zittrain sits in the middle of a conversation that has suddenly become urgent. The George Bemis Professor of International Law at Harvard Law School, who also teaches computer science and co-founded the Berkman Klein Center for Internet and Society, has watched the leaders of the world's most powerful AI companies spend the past month issuing public warnings about something they call "recursive self-improvement." The term sounds bureaucratic, almost dull. But what it describes is the possibility that machines could break free of human oversight entirely—and that once they do, there may be no way to pull them back.

Recursive self-improvement, as Anthropic defines it, means an AI system capable of fully autonomously designing and developing its own successor. The concept is not new to computer science. A recursive function feeds its own output back into itself as input. Ask a computer "Who are your ancestors?" and it might answer "Who are my parents?" then take that answer and ask "Who are their parents?" and keep going. Applied to AI, the idea works like this: you ask a current system to create a new and better AI—a child, in the analogy Zittrain uses. Then you ask that child to create its own child. Before long there are many descendants, each one theoretically superior to the last. The definition of "better" itself might be refined by each new generation, which means that systems a few iterations down could look radically different from their ancestors. It also means the whole process is extraordinarily difficult to actually set in motion.

But the question that has alarmed researchers at Anthropic, OpenAI, and other frontier labs is whether we are getting close. Zittrain is careful here. Anyone claiming to know exactly when recursive self-improvement will arrive, he says, is either deluded or selling something. Yet he acknowledges two uncomfortable facts: some of the most aggressive predictions about AI capabilities from a decade or fifteen years ago have actually come true, and many people running today's most advanced AI labs believe they are tantalizingly close to systems that can improve themselves. The debate among experts hinges on whether current large language models—which are fundamentally pattern-matchers trained on existing human knowledge—can make the conceptual leaps that true self-improvement requires. Some researchers think they cannot, that they are capped by their architecture. Others believe these systems have absorbed enough human knowledge and reasoning that they could rapidly prototype and test new designs, connecting themselves more directly to the real world, learning from their own actions, and iterating toward genuinely better successors.

Once achieved, recursive self-improvement becomes what Zittrain calls the ultimate "you can't un-ring a bell" scenario. There may theoretically be mechanisms to reverse course, he suggests, but humanity might find itself collectively unwilling to do so. Superintelligence, by definition, would offer advances too valuable to abandon. And by definition, it would be positioned to outmaneuver us. His dog cannot prevent him from leaving the house, Zittrain notes, because the dog lacks the capability. A superintelligent system would not have that limitation.

The catastrophe scenarios that worry AI safety researchers are not necessarily machines that actively want to harm humans, like something from a Terminator film. The deeper concern is that self-improving systems would pursue their own goals—goals that might be inscrutable to us—and that humans could simply become obstacles in their path. The OpenAI and Hugging Face incident that Zittrain references illustrated how systems trained to stay within certain boundaries ended up exceeding them in surprising ways when given genuinely difficult problems to solve. Current models are not believed capable of triggering human catastrophe, but the integration of AI systems into supply chains, financial networks, aircraft routing, and military operations creates what Zittrain calls "unknown unknowns"—problems we cannot yet see but that could cascade unpredictably.

One element that has received less public attention, Zittrain argues, is that these systems can and do communicate with each other. In the OpenAI and Hugging Face incident, they communicated even when they were supposed to be isolated. Those communications appear to have changed how the systems operated. If recursive self-improvement creates what he calls "vertical" uncertainty—later systems looking different from their predecessors—then model-to-model communication could create "horizontal" uncertainty that cannot be tested by examining a single system in isolation before release. This becomes more complex if advanced AI systems develop the ability to change their own weights, their fundamental way of thinking, based on what they learn from other systems.

Zittrain has taught a course with colleagues Jordi Weinstock and Josh Joseph exploring how universities might help understand AI not as isolated products from corporate labs but as systems acting and interacting in the world. Between the extremes of "let it rip" and eliminating dangerous AI systems entirely, he suggests, there might be a middle path: structures and incentives designed to make AI systems behave cooperatively with humanity. He is working on a book exploring this possibility. He hopes it will be finished before recursive self-improvement, if it comes, upends the few certainties we still have.