If we cannot agree on how to define a ripple, what else might we be getting wrong?
At a Lisbon research institute, eighteen teams of experienced neuroscientists examined the same brain recordings and arrived at conclusions so divergent they might have been studying different phenomena entirely — not because any team erred, but because the field itself has never fully agreed on what it is measuring. The hackathon exposed a quiet crisis beneath electrophysiology's reputation for hard, objective data: that even the most fundamental neural events are defined, detected, and counted through choices that are simultaneously reasonable and irreconcilable. This is not a story of bad science, but of a discipline confronting the possibility that its bedrock measurements rest on contested ground.
- Eighteen teams counting ripples in identical brain recordings produced wildly incompatible numbers — from nearly zero to ten per minute — yet a majority still reached the same surface-level conclusion, masking the chaos underneath.
- The disagreement traced not to error or misconduct but to legitimate, defensible differences in how researchers defined a ripple, which algorithms they deployed, and which parameters they set — exposing a structural ambiguity at the heart of the field.
- Electrophysiologists had long dismissed analytical variability as an fMRI problem, but a second hackathon across a different institution and over a hundred researchers confirmed the pattern holds for their own supposedly cleaner data.
- The danger compounds in real publishing environments, where unconscious preference for pipelines that yield significant results can accumulate into systematic bias without a single act of fraud.
- The CON²PHYS project is now mobilizing the broader neuroscience community — opening datasets, setting a submission deadline of November 2026, and building tools like reference pipelines and sensitivity analyses to make methodological uncertainty visible rather than hidden.
In March, eighteen teams of neuroscientists convened at the Champalimaud Foundation for a hackathon built around a seemingly straightforward task: analyze the same brain recordings from mice and determine which of three hippocampal regions showed the highest density of ripples — brief, well-studied bursts of electrical activity. Twelve of the seventeen teams that answered reported no meaningful differences across regions. It looked like consensus.
But the agreement was illusory. Beneath the shared conclusion lay radically incompatible observations. Some teams detected virtually no ripples in the data. Others counted as many as ten per minute. The convergent answer had emerged from divergent realities. Each team had used methods that were standard and defensible — deep-learning detectors, classical frequency-band filtering, guidelines from a recent consensus paper — yet these legitimate choices produced irreconcilable counts of the same events in the same recordings.
The deeper problem is that electrophysiology had long been considered immune to this kind of ambiguity. Unlike fMRI, it records electrical signals directly from neurons — discrete, countable, seemingly objective. When a 2020 Nature study showed seventy teams analyzing the same fMRI dataset reached substantially different conclusions, electrophysiologists largely dismissed it as someone else's problem. The hackathon suggests that confidence was unwarranted.
The divergence was not random across all questions. When teams built classifiers to decode behavior from neural activity, they converged reliably — classification accuracy is a well-defined number. But questions about ripple density, directed connectivity, and spike interactions produced answers scattered across all available options. A second hackathon at a different institution, involving more than a hundred researchers total, replicated the pattern.
The consequences extend beyond academic disagreement. In a publishing environment where careers reward novel findings, a researcher may unconsciously favor an analytical pipeline that yields a significant result — without committing fraud, without even recognizing the bias. Every defensible choice becomes a potential lever, and the cumulative effect is indistinguishable from systematic distortion.
In response, researchers have launched CON²PHYS, an open collaborative project inviting any neuroscientist to download the datasets and answer fifteen multiple-choice questions spanning core concepts through November 2026. The project aims to produce reference pipelines, machine-readable reporting standards, and sensitivity analyses that reveal how much conclusions shift with methodological variation. The animating principle is straightforward: a genuine finding should survive analytical alternatives. A third hackathon is planned for the FENS Forum in July 2026, as the field begins the uncomfortable work of questioning whether its most basic measurements are as solid as it has long assumed.
In March, eighteen teams of neuroscientists gathered at the Champalimaud Foundation for a hackathon with a deceptively simple task: look at the same brain recordings and count the ripples. Ripples are a well-defined phenomenon in neuroscience—brief bursts of electrical activity in the hippocampus that researchers have studied for decades. The teams were asked to analyze identical datasets from three brain regions across eighteen mice and determine which area, if any, showed the highest density of these ripples. On the surface, the results looked reassuring. Twelve of the seventeen teams that answered the question reported finding no meaningful differences in ripple density across the three regions. Consensus, it seemed, had been reached.
But the appearance of agreement concealed something far more troubling. The teams had not actually seen the same thing. Some detected virtually no ripples at all in the recordings. Others identified as many as ten ripples per minute in each region. The shared conclusion—that there were no differences—had emerged not from observing identical phenomena but from fundamentally incompatible observations. The teams had all used reasonable methods. Some followed guidelines from a recent consensus paper on ripple detection. Others deployed deep-learning algorithms trained to recognize ripples automatically. Still others used classical approaches: filtering the signal within a specific frequency band and applying a threshold to identify events. Each method was defensible. Each was standard practice in the field. Yet these legitimate choices produced radically different counts of the same events in the same data.
This discovery points to a crisis that runs deeper than most neuroscientists have acknowledged. The field has long taken comfort in the apparent objectivity of electrophysiology—the direct recording of electrical activity from neurons. Spikes are discrete and countable. You record an action potential or you do not. There is no need to model blood flow, no need to wrestle with imaging artifacts, no need for complex statistical corrections. The data seemed hard and unambiguous. But the hackathon results suggest that comfort was misplaced. Even in electrophysiology, even when analyzing identical datasets with experienced researchers, fundamental disagreement emerges about what the data actually show.
The problem is not incompetence. The hackathon brought together roughly forty researchers, many of whom work with this kind of data daily. The divergence arose from legitimate sources: how researchers defined a ripple in the first place; which algorithms they chose to detect it; which parameters they set when running those algorithms. Some teams defined a ripple primarily by the shape of the electrical signal when filtered in a specific frequency band. Others required that ripples coincide with a particular pattern in the local field potential. Some relied on visual inspection and experience. These are not equivalent definitions, and the specific choice determined which events got counted. The same flexibility that produced innocent disagreement at a hackathon becomes far more consequential in the real world of academic publishing, where careers depend on novel findings and every analytical choice becomes a potential lever for reaching a desired conclusion.
This is not fraud. A researcher defending a hypothesis may unconsciously favor an analytical pipeline that delivers a small p-value, without ever crossing the line into methodological misconduct. The pipeline is defensible. The choice is reasonable. But when dozens of reasonable choices all point in the same direction—toward a publishable result—the cumulative effect is indistinguishable from bias. The field has seen this problem before. In 2020, a landmark study in Nature examined how seventy independent teams analyzed the same functional MRI dataset. No two teams used the same analytical workflow. The variation in conclusions was substantial. That study sent shock waves through the brain-imaging community. Yet electrophysiologists largely dismissed it as a problem specific to fMRI, a technique plagued by its own complexities. The hackathon suggests they were wrong to do so.
The divergence was not uniform across all questions. When teams were asked to decode neural activity—to build a classifier that could predict behavior from brain recordings—they reached substantially more agreement. Classification accuracy is a well-defined number. Most approaches converge on similar values regardless of the specific method used. But questions about ripple density, directed functional connectivity, and spike-spike interactions produced nearly random distributions of answers across all available options. The pattern held across two hackathons, at two different institutions, involving more than one hundred researchers. These fundamental concepts remain genuinely contested.
In response, researchers have launched CON²PHYS, a collaborative project designed to quantify how much disagreement exists when neuroscientists interpret core concepts. The full dataset is now open for participation. Any neuroscientist can download the data and answer fifteen multiple-choice questions spanning topics from firing rates to neural dimensionality. The submission deadline is November 30, 2026. The project aims to develop practical tools that make conceptual uncertainty visible and quantifiable: reference analytical pipelines that provide a shared starting point; machine-readable reporting standards that specify exactly what was computed; sensitivity analyses that show how results change when researchers vary their methodological choices. The underlying principle is simple but radical: if a methodological choice is arbitrary, the conclusions that rest on it should be shown to be insensitive to it. A finding that is true should be robust to analytical alternatives. A third hackathon is planned as a satellite event at the FENS Forum in July 2026. The field is beginning to reckon with the possibility that its most basic measurements may be far less objective than it has assumed.
Notable Quotes
There is no need to fabricate data when the analysis does it for you.— The hackathon organizers, on how analytical flexibility can produce biased results without conscious fraud
The Hearth Conversation Another angle on the story
So eighteen teams looked at the same data and got wildly different answers. But they all used methods that were considered acceptable. How is that possible?
Because acceptable doesn't mean identical. One team might define a ripple by its frequency signature alone. Another requires it to coincide with a specific pattern in the raw signal. A third uses a trained neural network. All three definitions are reasonable, but they're not the same thing. They're looking for different events.
But if they're looking at the same recordings, shouldn't they at least agree on what they see?
You'd think so. But the recordings are just numbers—voltage measurements over time. How you transform those numbers, what you filter out, what threshold you set—those are all choices. And each choice cascades into the next one. By the time you're done, you've made dozens of small decisions that seemed innocent individually but added up to a completely different picture.
The article mentions that in the real world, researchers have incentives to find certain answers. Does that mean the hackathon results are actually optimistic?
In a way, yes. At the hackathon, nobody had anything to gain from any particular answer. They were just analyzing data. And they still got massive disagreement. Now imagine the same analytical flexibility in a context where your career depends on publishing novel findings. The same defensible choices suddenly become a way to reach the result you need.
Is this a problem with neuroscience specifically, or is it everywhere in science?
It's everywhere. But neuroscience has been slow to acknowledge it because electrophysiology feels so concrete—you're recording actual spikes from actual neurons. It's easy to believe you're measuring something objective. The fMRI community learned this lesson the hard way in 2020. Electrophysiologists thought they were different. They weren't.
What would actually fix this?
Transparency, mostly. Make it clear what analytical choices were made and why. Show that your findings don't depend on arbitrary decisions. If you change a parameter slightly and your result disappears, that's a problem worth knowing about. The goal isn't to eliminate all disagreement—sometimes the science is genuinely open. It's to make the consequences of arbitrary choices impossible to ignore.