The maths bust-up fuelling fears that AI is undermining science

In a former watch factory in central London, a bank of white freezers hums away. Inside are dozens of small boxes of barcoded glass tubes holding, between them, one hundred billion genes. Dr Oliver Vince, co-founder of the facility, says just two of those boxes may contain ten times more information than all the text on the internet. His firm, Basecamp Research, intends to gather as much DNA as it can to build an AI that speaks the language of biology, much as US and Chinese tech companies scooped up online writing to build the Large Language Models behind today's AI systems.

Vince is one of many in business and academia who expect AI to supercharge research. Increasingly, though, that expectation arrives with controversy. Last month OpenAI, the maker of ChatGPT, said its AI agents had solved, in 88 hours, a major unsolved maths problem tied to the Navier-Stokes equations. The body that runs a prize for the problem said it had apparently been settled, but that its review would take time.

Hours before the announcement, Tristan Buckmaster, a mathematician at New York University, went public with concerns. He had formed an independent team with Levent Alpöge, a mathematician who works for OpenAI's rival Anthropic. They were pursuing the same goal and kept their drafts in OpenAI's coding tool, Codex. Buckmaster said his co-researcher was tipped off that news of their progress had reached OpenAI, though he did not say how. He asked whether work stored in Codex could have fed into OpenAI's own, while stressing he was not accusing anyone of anything.

OpenAI says nobody at the company, human or AI, saw the other team's work before publication. At first it said that, while unlikely, it could not rule out that anonymised data from use of its products might have helped improve its models. Days later it said Buckmaster's Codex prompts from the two months before the announcement could not have influenced the system in any way. As the BBC puts it, we still don't fully know what happened.

The episode points to a quieter problem than the usual fears of engineered viruses: that AI could erode trust in science, by casting doubt on the novelty and accuracy of research. Chatbots are known to hallucinate films that never existed or quotes never spoken, and there is now evidence the same is happening in science. Dalmeet Singh Chawla, a journalist who has covered research misconduct for a decade, says generative AI has disrupted the whole scientific process. He argues journals' integrity staff must grow drastically in number and effort, and says that right now they do not appear to be.

The case for AI is strong. Scientists handle data on a scale no brain can absorb, from telescopes to particle accelerators. In 2020 Google DeepMind's AlphaFold predicted 200 million protein folds in a year, work that would have taken hundreds of millions of years in laboratories. The results were given free to researchers, and DeepMind's head and the project's lead scientist won the Nobel Prize for Chemistry. Dr Pushmeet Kohli, who leads DeepMind's science work, calls it a watershed moment.

Then came a warning. A DeepMind model produced what the firm said were more than two million new crystal structures, billed as nearly 800 years' worth of knowledge, with about 380,000 judged promising enough to make in a lab. Chemists Anthony Cheetham and Ram Seshadri of the University of California sampled them and wrote in Chemistry of Materials that they had yet to find any strikingly novel compounds, calling many trivial adaptations of known materials. DeepMind rejects this, saying it tested each compound for novelty and that swapping one atom is how many useful materials emerged.

The larger worry is AI slop in journals. Prof Dmitry Kobak of Ghent University searched more than a million biomedical papers for words favoured by LLMs and found that by December last year nearly 90% bore AI fingerprints, though detection tools are imperfect and have flagged even the US Constitution. At Stanford, Weixin Liang and James Zou found up to 17% of referee reports at four major computer science conferences were substantially modified by a language model. A Cornell team found AI-compiled citations included around 147,000 phantom pieces of research last year.

There is hope too. Prof Andrea Pauli of the Vienna BioCenter spent almost ten years on how a sperm enters an egg; with responsible use of AI, her lab found a zebrafish egg surface protein pointing to a lock-and-key mechanism. Vince, who recovered from leukaemia soon after founding Basecamp, says he sees a technically feasible path to curing cancer. It is a direction, not a promise, and the open question is whether AI carries science there or wears away its foundations.