Efficient AI Model Identifies Gaps in State HPV Vaccination Information Online

Researchers have developed a streamlined artificial intelligence system that can rapidly evaluate how state health departments communicate information about HPV vaccination online—work that could reshape how public health agencies monitor and improve their digital messaging.

The problem is straightforward but consequential. Human papillomavirus causes roughly 48,000 cancers annually in the United States, yet vaccination rates lag behind what they should be. Only 76.8 percent of American adolescents and 39.9 percent of adults aged 18 to 26 have received the HPV vaccine, compared to 82 percent of 12-year-old girls in Australia, a country on track to eliminate cervical cancer within two decades. State health department websites serve as trusted sources where people turn for authoritative information about vaccination. The quality and completeness of what those websites say matters—it shapes awareness and influences decisions. Yet no one had systematically evaluated what 48 different state health departments were actually telling the public about HPV.

A team led by researchers at PLOS Digital Health tackled this by building a smaller, faster artificial intelligence model trained to read and categorize health information. They started by collecting 400 paragraphs from state health department websites and having human annotators label them with 15 key pieces of information—things like whether the site explained that HPV has no cure, how many vaccine doses are required, whether the vaccine prevents cancer, and whether it's recommended for both males and females. This annotated dataset became the training ground.

Then came the technical innovation. The researchers used a technique called knowledge distillation, where a large, powerful language model (Llama 3.1 70B) was trained to make accurate judgments about the health content, and those judgments were then transferred to a much smaller model (RoBERTa Large) that could do the same work far more efficiently. The smaller model achieved an F1 score of 0.74 on test data—nearly matching the larger model's 0.77 score. More strikingly, it processed information 350 times faster. Where the large model would have taken three weeks to analyze the full corpus of state websites using three high-end graphics processors, the smaller model could do it in hours.

When the researchers deployed this efficient model across all 48 state health department websites, analyzing more than 14,000 paragraphs, a clear picture emerged: coverage was wildly uneven. Five states—New Mexico, New York, North Dakota, Texas, and Washington—included all 14 of the key information points the researchers were looking for. Thirty-one states covered at least 10 topics. But five states offered fewer than five key pieces of information. No state covered all 15 categories the researchers had defined, and no single topic was universally absent, suggesting that gaps were scattered across the country rather than concentrated in any one area.

The research also revealed something unexpected about how the two models performed differently. The larger model excelled at extracting straightforward, explicitly stated facts—it achieved perfect scores on identifying whether sites said HPV has no cure or how many doses are needed. The smaller model, by contrast, performed better at understanding information embedded within broader narratives or presented indirectly. When a website said something like "These are all viruses that can be passed during sex," the smaller model was more likely to correctly infer that the site was discussing sexual transmission. This suggests that model compression through knowledge distillation may preserve or even enhance the ability to interpret implicit information while reducing performance on simple fact extraction.

The implications extend well beyond HPV. The researchers argue that this framework could be adapted to monitor how state health departments communicate about influenza, COVID-19, chronic disease prevention, or emerging public health threats. It could enable real-time tracking of information quality across digital ecosystems, allowing public health agencies to identify gaps quickly and respond with improved messaging. The system could also help assess whether information is being presented consistently across different language versions of state websites—a question the current study did not address but flagged as important for future work, given the linguistic diversity of the United States.

The researchers acknowledged limitations. Their analysis focused only on text from state health department websites, missing information in PDFs, videos, and interactive media. They examined only English-language content, leaving open questions about whether multilingual versions of these sites present information equally well. And they relied on Google's search algorithm to surface relevant pages, which worked well for most states but theoretically could have missed content on states where HPV information was scattered across many pages. Despite these constraints, the work provides the first systematic assessment of how state governments are communicating about HPV vaccination online—and a practical tool for doing similar assessments at scale.