A Maths Feud and Phantom Citations Raise Alarm Over AI's Growing Role in Science

In a former watch factory in central London, a bank of white freezers hums through the day. Inside are dozens of small boxes of barcoded glass tubes holding some one hundred billion genes. Dr Oliver Vince, co-founder of the facility, says just two of those boxes may contain ten times more information than all the text on the internet. His company, Basecamp Research, wants to gather as much DNA as it can and build an AI that speaks the language of biology, much as US and Chinese tech firms scraped written material online to build the large language models behind today's AI systems.

Vince is one of many in business and academia who expect AI to supercharge research. But the promise increasingly arrives with controversy. Last month, OpenAI said its AI agents had solved, in 88 hours, a major unsolved problem in maths connected to the Navier–Stokes equations. The body that runs a prize for the problem said it had apparently been settled, but that its review would take time.

Hours before that announcement, Tristan Buckmaster, a mathematician at New York University, went public with concerns. He had formed an independent team with Levent Alpöge, who works for OpenAI's rival Anthropic. They were pursuing the same goal and kept their drafts in OpenAI's coding tool, Codex. Buckmaster said Alpöge was tipped off that news of their progress had reached OpenAI, though he did not say how. He asked whether work stored in Codex could have fed into OpenAI's own, while stressing he was not accusing anyone of anything.

OpenAI said nobody at the company, human or AI, saw the other team's work before it was published. At first it said that, while unlikely, it could not rule out that anonymised data from use of its products had helped improve its models. Days later it said Buckmaster's Codex prompts from the two months before the announcement could not have influenced the system in any way. As the BBC puts it, it is a complicated story, and what happened is not fully known.

The episode points to a quieter risk than the usual fears of engineered viruses: that AI could erode trust in the scientific process, and in the novelty and accuracy of research. There is also hallucination, the tendency of chatbots to invent films or quotes, which there is now evidence is spreading into science. Dalmeet Singh Chawla, a science journalist who has spent a decade covering research misconduct, says generative AI has disrupted the whole scientific process. He argues publishers' integrity staff will need to grow drastically in number and effort, and says that right now they do not appear to be doing so.

The case for AI in science is real. Google DeepMind's AlphaFold predicted 200 million protein folds in a year, work that would have taken hundreds of millions of years in laboratories. The results were given free to researchers, and DeepMind's head and the project's lead scientist won the Nobel Prize for Chemistry. But other claims have drawn fire. DeepMind announced more than two million new crystal structures, described as nearly 800 years' worth of knowledge, of which about 380,000 looked promising enough to make in a lab. Chemists Anthony Cheetham and Ram Seshadri at the University of California sampled them and, writing in Chemistry of Materials, said they had yet to find any strikingly novel compounds, calling many trivial adaptations of known materials. DeepMind rejects this, saying it tested each compound for novelty and that swapping a single atom is how many useful materials arose.

The larger worry is AI slop in journals. Prof Dmitry Kobak of Ghent University searched more than a million biomedical papers for words favoured by language models and found that by December last year nearly 90% bore AI fingerprints, though detection tools are imperfect and have even flagged the US Constitution. A Stanford team led by Weixin Liang and James Zou found up to 17% of text in referee reports at four major computer science conferences had been substantially modified by a language model. A Cornell team found AI-compiled citations included around 147,000 pieces of phantom research last year.

Publishers are responding. Elsevier has developed about 100 signals that flag papers for human scrutiny. Nature's journals ban AI-generated images and tell referees never to upload manuscripts into AI systems, and arXiv bans researchers for a year if they submit invented references. DeepMind's Dr Pushmeet Kohli says abandoning AI would be like computing without calculators, but insists someone must take accountability for what an agent does. Prof Andrea Pauli of the Vienna BioCenter offers a hopeful case: after almost ten years on how sperm enters an egg, her lab, using AI responsibly, found a protein on zebrafish eggs pointing to a lock-and-key mechanism. Vince, who recovered from leukaemia, says there is a technically feasible path to curing cancer. The open question is whether AI carries science there or wears away its foundations.