We're essentially flying blind with these systems.
For all the intelligence artificial minds now display, the reasons behind their learning have remained opaque — a gap that troubles both scientists and the engineers who build these systems. A team of Harvard physicists has moved to close that gap, constructing a simplified mathematical model drawn from statistical physics to illuminate why neural networks learn as well as they do without collapsing under the weight of their own complexity. Their work suggests that the very high-dimensional chaos inside large AI systems may be what steadies them — that disorder, at scale, becomes a kind of order. It is, perhaps, the beginning of a theory of gravity for the age of artificial minds.
- Millions rely daily on AI systems whose inner workings remain fundamentally unexplained — a black box at the center of modern technological life.
- The puzzle of overfitting haunts the field: by all conventional reasoning, models as large and complex as ChatGPT should memorize rather than learn, yet they don't.
- Harvard physicists borrowed the tools of statistical physics — specifically renormalization theory — to build a toy model simple enough to solve on paper but rich enough to mirror real network behavior.
- Their findings suggest that high-dimensional statistical fluctuations don't corrupt learning but stabilize it, offering a potential mechanism behind one of AI's most persistent mysteries.
- The research points toward a future of more efficient, less energy-hungry AI — but only if the field can move from empirical observation to genuine theoretical understanding.
The AI systems millions use every day operate as black boxes: we know they work, but not why. A team of Harvard physicists has taken a meaningful step toward an answer, using tools from statistical physics to build a simplified mathematical model of how neural networks learn.
Lead author Alexander Atanasov draws an analogy to the history of science itself. Kepler identified the patterns of planetary motion long before Newton explained the force behind them. AI research may be in a similar pre-Newtonian phase — empirical laws exist describing how larger models with more data tend to perform better, but no unifying theory explains the mechanism. That gap has real consequences: today's AI systems consume enormous energy, and a deeper understanding could change that.
Because analyzing a full-scale neural network with exact mathematics remains impossible, the team worked with a toy model based on ridge regression — simple enough to solve analytically, yet capable of reproducing key behaviors seen in real networks. Chief among those behaviors is the mystery of overfitting: large models, by conventional theory, should memorize training data rather than generalize from it. They largely don't, and no one has fully explained why.
The Harvard study proposes an answer rooted in renormalization theory. In the extraordinarily high-dimensional spaces where AI systems operate, small random variations naturally arise — and rather than destabilizing learning, these fluctuations appear to steady it. Many microscopic details collapse into a small number of governing parameters, allowing complex systems to behave with surprising simplicity at scale.
Senior author Cengiz Pehlevan suggests this mechanism may broadly explain why over-parameterized networks avoid overfitting. Colleague Jacob Zavatone-Veth notes the toy model can serve as a baseline for understanding learning across many different high-dimensional systems. In identifying which aspects of learning are generic and which are specific, the work edges the field closer to its own theory of gravity.
The artificial intelligence systems that millions of people interact with every day—ChatGPT, Claude, DeepSeek, Gemini—operate as black boxes. We know they work. We don't know why. A team of physicists at Harvard University has now taken a step toward cracking that mystery by building a simplified mathematical model of how neural networks learn, using tools borrowed from statistical physics to peer inside the machinery.
The approach echoes a moment from the history of science itself. Johannes Kepler observed the motion of planets and discovered mathematical patterns—scaling laws relating orbital periods to distances from the sun—without understanding the mechanism that produced them. Isaac Newton came later and provided the deeper explanation: gravity. Today's artificial intelligence research may be in a similar phase, according to Alexander Atanasov, a PhD student at Harvard and lead author of the new study, published in the Journal of Statistical Mechanics: Theory and Experiment. Researchers have identified empirical laws describing how neural networks behave—larger models with more data tend to perform better—but they lack a unifying theory explaining why. This gap matters. Current AI systems consume staggering amounts of energy. A deeper understanding of learning mechanisms could lead to more efficient designs.
Neural networks, Atanasov explains, are not hand-written algorithms. They are more like organisms grown in a laboratory. The networks consist of many simple processing units called artificial neurons, each performing basic operations but connected in intricate webs. From this networked structure emerges behavior that looks intelligent. Yet as the number of components grows, predicting what the system will actually do becomes exponentially harder. The whole is far more than the sum of its parts.
Because analyzing a full-scale neural network with exact mathematics is currently impossible, Atanasov and his colleagues chose to work with a toy model—a simplified system that preserves key features of real networks while remaining mathematically tractable. Their model is based on ridge regression, a variant of linear regression used to estimate relationships between variables. It is simple enough to solve on paper. It is also complex enough to reproduce several phenomena observed in large neural networks.
One of those phenomena is the great mystery of overfitting. When a neural network trains on large datasets, it can memorize the data rather than learning patterns that generalize to new information. Counterintuitively, this does not happen as often as theory predicts. Large models often learn without overfitting. Performance improves as more training data is added. This should not work, yet it does.
The Harvard study offers a possible explanation rooted in renormalization theory, a framework from statistical physics. The key lies in dimensionality. When AI systems like ChatGPT process information, they operate in spaces with thousands or millions of variables—far more than the simple two-variable example of predicting height from weight. In such high-dimensional spaces, small random variations naturally emerge. Renormalization theory shows that many microscopic details can be absorbed into a small number of parameters, allowing complex systems to display relatively simple large-scale behavior. Using their toy model, the researchers demonstrate how these high-dimensional fluctuations can actually stabilize learning rather than destabilize it.
Cengiz Pehlevan, an associate professor of applied mathematics at Harvard and senior author, suggests the same mechanism may explain why current neural networks avoid overfitting despite being vastly over-parameterized. Jacob Zavatone-Veth, a junior fellow at Harvard, notes that the simplified model could serve as a baseline for understanding learning in high-dimensional systems generally. By studying a model simple enough to analyze mathematically, researchers can identify which aspects of learning are generic—likely to appear across many different networks—and which depend on specific details. In this way, studies like this may help clarify fundamental principles underlying learning in complex systems, bringing the field closer to its own theory of gravity.
Citações Notáveis
Deep learning models are not algorithms written by hand. They're much more similar to an organism being grown in a lab.— Alexander Atanasov, Harvard PhD student and lead author
Despite being extremely large, these models can learn from the data without overfitting. This is one of the great mysteries of deep learning.— Alexander Atanasov
A Conversa do Hearth Outra perspectiva sobre a história
Why does it matter that we don't understand how neural networks work? They're already producing useful results.
Because we're essentially flying blind. We can make them bigger and feed them more data, but we don't know why that works. It's like having a car that runs but no understanding of engines—you can't fix it when it breaks, and you can't make it better.
So this toy model—ridge regression—how does that help us understand ChatGPT?
It doesn't directly. But it's simple enough that we can solve it mathematically and see what's actually happening. If we find principles at work in the toy model, we can ask whether those same principles might be operating in the real thing.
You mentioned overfitting. Why is it surprising that large models don't overfit?
Because intuitively, a bigger model should be more prone to memorizing its training data rather than learning general patterns. But empirically, that's not what happens. Bigger models with more data actually generalize better. It's backwards from what the theory predicts.
And your explanation involves something called renormalization theory?
Yes. In very high-dimensional spaces—thousands or millions of variables—random fluctuations naturally appear. Renormalization theory shows that these fluctuations can actually stabilize learning. The noise becomes a feature, not a bug.
What happens next? Does this lead to better AI systems?
That's the hope. If we understand the fundamental mechanisms, we could design systems that are more efficient and reliable. Right now, training these models consumes enormous energy. Understanding why they work might let us do more with less.