Self-harm history matters too much to stay buried
Within the vast architecture of modern medical recordkeeping, a quiet crisis of invisibility persists: the histories most critical to preventing suicide are often the ones least visible to the systems designed to track them. Researchers at the University of New Mexico have used machine learning to measure this gap among 1.3 million veterans, finding that standard diagnosis codes capture only a quarter of documented self-harm histories — a fourfold undercount with profound consequences for clinical care and public health planning. The study is less a technological triumph than a reckoning with the distance between what is written and what is seen, between what is known and what is acted upon.
- A fourfold gap between coded and actual self-harm prevalence — 1.85% versus 7.9% — means health systems are planning mental health services around a dramatically incomplete picture of need.
- Even when self-harm made it into diagnosis codes, it was absent from problem lists — the most visible summary fields clinicians consult — in more than three-quarters of cases, compounding the invisibility at the point of care.
- Self-harm history is among the strongest predictors of future suicide risk, so every undetected case represents a missed opportunity for preventive intervention in a population already at elevated risk.
- The PULSNAR machine learning method was purpose-built for the uncertainty of real medical data, estimating hidden prevalence among uncoded records rather than treating absence of a code as absence of a condition.
- The research team is now extending the method to PTSD, depression, bipolar disorder, and sleep disorders, suggesting the self-harm finding may be one instance of a systemic pattern of under-recording across mental health.
A patient's medical record can run to half a million lines of clinical notes — and buried somewhere inside those lines, for a significant share of patients, is a history of self-harm that the record's own summary systems never surface. A new study from the University of New Mexico School of Medicine has put a number on that invisibility: standard diagnosis codes capture only about one-fourth of the self-harm histories actually documented in veteran charts.
Analyzing electronic health records for more than 1.3 million veterans in the Veterans Health Administration, the research team estimated that roughly 7.9 percent of patients had documented self-harm history — compared to the 1.85 percent that appeared when researchers looked only at diagnosis codes. The gap compounds further in problem lists, the summary sections where providers flag major health conditions: even among veterans who did have a self-harm diagnosis code, fewer than one in four had the condition listed there, the place a care team would most naturally look during a visit.
The stakes are high. Self-harm history is one of the strongest predictors of future self-harm and suicide risk, and it shapes clinical thinking about depression, PTSD, bipolar disorder, substance use, and traumatic brain injury. Missing that history means missing critical context at the moment treatment decisions are made.
To uncover these hidden patterns, the team used a machine learning method called PULSNAR — Positive Unlabeled Learning Selected Not At Random — designed specifically for the messy reality of medical data, where a missing code doesn't mean a condition was absent, only that it wasn't recorded. The method estimates how many uncoded patients resemble those who were coded, accounting for the uneven likelihood of documentation across different cases.
Led by professor Christophe Lambert, the team framed the problem as one of systems-level visibility rather than individual clinical failure: no clinician can read hundreds of thousands of notes during a routine visit. The method is still a research tool, not yet ready for direct clinical deployment, but the team is already extending it to opioid use disorder, PTSD, depression, bipolar disorder, and sleep disorders — conditions that may share the same quiet pattern of being known but unseen.
A patient's medical record can be enormous—some of the ones researchers reviewed contained more than half a million lines of clinical notes. Inside those records, crucial information about self-harm often sits buried, invisible to the systems that hospitals and health systems use to count and track mental health conditions. A new study from the University of New Mexico School of Medicine has quantified just how invisible: diagnosis codes, the standardized labels clinicians use to document what they've treated, captured only about one-fourth of the self-harm histories that were actually written down in patient charts.
The research team analyzed electronic health records for more than 1.3 million veterans served by the Veterans Health Administration. Using a machine learning method designed to work with the messy, incomplete reality of medical data, they estimated that documented self-harm was present in roughly 7.9 percent of those patients—more than four times the 1.85 percent that showed up when researchers looked only at diagnosis codes. The gap matters because it shapes everything downstream: how researchers measure the scope of mental health need, how health systems plan services, and whether clinicians have the full picture of a patient's history when making treatment decisions.
The problem runs deeper than just diagnosis codes. Problem lists—the summary sections where providers are supposed to flag a patient's major health conditions—showed another visibility gap. Even among veterans who had a diagnosis code for self-harm, only about 22.6 percent had self-harm or a history of self-harm listed on their problem list. This means that even when the condition made it into the formal coding system, it often remained absent from one of the record's most visible summary fields, the place a care team would most naturally look.
Why does this matter clinically? Self-harm history is one of the strongest predictors of future self-harm and suicide risk. It also shapes how clinicians think about and treat depression, PTSD, bipolar disorder, substance use, and traumatic brain injury—conditions that frequently occur alongside self-harm. Missing that history means missing a crucial piece of context for understanding a patient's mental health needs.
The researchers used a novel machine learning approach called PULSNAR—Positive Unlabeled Learning Selected Not At Random—to uncover these hidden patterns. Most machine learning methods require clear examples of both positive and negative cases. But medical records don't work that way. A missing diagnosis code doesn't prove a patient never had the condition; it might just mean the clinician didn't code it. PULSNAR was built for exactly this kind of uncertainty. It learns from patients who do have a code, then estimates how many similar patients might be present among those without a code, accounting for the fact that some cases are more likely to be coded than others.
The team, led by Christophe Lambert, a professor at UNM's School of Medicine, emphasized that this is fundamentally a systems-level visibility problem. No clinician can be expected to read through hundreds of thousands of lines of notes during a routine visit. The machine learning method doesn't try to predict whether any individual patient will harm themselves in the future. Instead, it flags patterns in the structured data that suggest self-harm history might be present but missing from the diagnosis codes, then compares those estimates against expert review of the actual clinical notes.
The researchers also noted that the Veterans Health Administration already uses specialized suicide and overdose reporting tools and doesn't rely solely on diagnosis codes to monitor suicide risk. This study asked a different question: How much documented self-harm history remains invisible in the parts of the medical record that researchers, care teams, and health systems can most easily search and quantify at scale? The answer suggests a significant blind spot in how mental health conditions are measured and tracked across large populations.
The work is part of a broader research program using the same machine learning approach to find other conditions that may be under-recorded in standard medical data. The team has already published related work on opioid use disorder and is extending the method to unrecognized PTSD, depression, bipolar disorder, and sleep disorders. Lambert noted that while the method is still a research tool and not ready for direct clinical use, it could eventually help health systems better estimate the true prevalence of under-recorded mental health conditions and identify records that warrant closer review. As he put it, self-harm history matters too much to remain buried in records that are impractical to review line by line during routine care.
Citas Notables
If we only count what is easy to see in diagnosis codes, we may substantially underestimate the need for mental health services.— Christophe Lambert, University of New Mexico School of Medicine
This is a systems-level visibility problem. The record can be enormous. No clinician can be expected to read all of that during a normal visit.— Christophe Lambert
La Conversación del Hearth Otra perspectiva de la historia
Why does it matter if diagnosis codes miss three-quarters of self-harm cases? Aren't clinicians still seeing the patients?
They're seeing the patients, yes, but they might not be seeing the full picture. If a clinician doesn't know about past self-harm—because it's not flagged in the problem list or the diagnosis codes—they might miss a critical risk factor for suicide. And at the systems level, if health administrators only count what's in the codes, they'll massively underestimate how many people need mental health services.
So this is about visibility, not about missing care?
It's both. The care might be happening, but the history is invisible. A clinician might treat someone's depression without realizing they have a self-harm history, which would change how they approach treatment. And researchers studying mental health outcomes will get the wrong numbers if they only count coded cases.
Why is self-harm history so hard to find in the first place?
Partly because clinicians write it in narrative notes—the long text sections of the record—but don't always code it formally. Partly because the record is enormous. Some veteran files had over 500,000 lines. No one can read all of that. And partly because there's no system forcing it into the visible summary fields.
What does the machine learning actually do differently?
It works with uncertainty. Most machine learning needs clear yes-and-no examples. But in medicine, a missing code doesn't mean the condition didn't happen—it might just mean it wasn't coded. This method learns from the cases that were coded, then estimates how many similar cases might be hiding in the uncoded group.
Is this ready to use in hospitals now?
Not yet. It's a research tool. But the researchers think it could eventually help hospitals identify which records need closer human review, and give them a more accurate count of how much mental health need actually exists in their population.