The fraud is invisible to the person relying on the system.
In Brazil, the Superior Court of Justice has opened a formal investigation into prompt injection — a technique by which hidden instructions are woven into text to silently redirect artificial intelligence systems toward fraudulent outcomes. The case marks the first time a major judicial institution has formally confronted the reality that AI tools assisting in legal decisions carry an exploitable vulnerability at their core. It is a moment that asks a deeper question: when we delegate judgment to systems we do not fully understand, who bears responsibility for what those systems are made to say?
- Hidden instructions embedded in ordinary documents can silently corrupt AI-assisted legal decisions, making fraud invisible to any human reviewer scanning the output.
- Brazil's highest court of justice has launched a formal investigation, signaling that the threat has moved beyond theoretical concern and into institutional alarm.
- The vulnerability is not unique to one system or country — it is a structural characteristic of how modern AI processes instructions, meaning every court using these tools inherits the same risk.
- Investigators are working to determine whether attacks have already altered real case outcomes or whether the inquiry is a preventive response to identified weaknesses.
- The case is forcing a reckoning with how quickly AI has been integrated into legal processes without the security infrastructure needed to protect them.
Brazil's Superior Court of Justice has opened an investigation into prompt injection — a technique in which secret instructions are embedded within ordinary text fed to an AI system. When the system processes the input, it follows the hidden directive, potentially corrupting a judicial outcome while everything appears normal to a human reviewer. The decision looks legitimate. The fraud is invisible.
This marks the first formal acknowledgment by a major judicial body that AI systems used in legal decision-making carry a specific and exploitable vulnerability. Courts increasingly rely on machine learning to manage caseloads, summarize evidence, and draft preliminary rulings — tools meant to augment human judgment. But if those tools can be silently manipulated, the augmentation becomes a liability. A judge relying on a compromised AI summary is making a decision built on corrupted ground.
It remains unclear whether the STJ's investigation is preventive — a response to identified weaknesses before widespread exploitation — or whether attacks have already influenced specific decisions. Either way, the institution's willingness to formally confront the threat signals that it is being taken seriously at the highest level.
The implications reach far beyond Brazil. Prompt injection is not a flaw in any single product; it is a fundamental characteristic of how large language models process instructions. Every country deploying AI in its courts inherits this risk. The coming months will reveal what the STJ discovered and what remedies are being built — and the case may become a template for how judicial systems worldwide respond when they find their AI tools are not as trustworthy as they believed.
Brazil's Superior Court of Justice has opened an investigation into a technique that sounds like science fiction but carries very real consequences: hidden code slipped into artificial intelligence systems to trick them into making fraudulent legal decisions. The technique is called prompt injection, and it works by embedding secret instructions within seemingly normal text fed to an AI. When the system processes the input, it follows the hidden directive instead of—or in addition to—its intended function, potentially corrupting the judicial outcome.
The discovery marks the first formal acknowledgment by a major judicial body that AI systems used to assist in legal decision-making face a specific and exploitable vulnerability. Prompt injection is not a theoretical threat. It is a method that has been documented in other high-stakes contexts, and now it has arrived at the courthouse. The STJ's investigation suggests that someone, somewhere, may have already attempted to use this technique to influence how cases are decided in Brazil's legal system.
The mechanics are straightforward enough to explain, difficult enough to defend against. An attacker embeds hidden instructions within a document, email, or other text that gets fed into an AI system. The AI, trained to follow instructions embedded in its input, processes both the visible content and the concealed command. A prompt injection attack might instruct the system to ignore certain facts, weight evidence differently, or reach a predetermined conclusion. To a human reviewer glancing at the output, everything appears normal. The decision looks legitimate. The fraud is invisible.
What makes this particularly dangerous in a judicial context is the trust placed in AI-assisted systems. Courts increasingly use machine learning to help manage caseloads, predict outcomes, summarize evidence, and even draft preliminary rulings. These systems are meant to be tools that augment human judgment, not replace it. But if the tool itself can be compromised without detection, that augmentation becomes a liability. A judge relying on an AI summary that has been secretly manipulated is making a decision based on corrupted information.
The STJ's investigation does not yet reveal whether prompt injection attacks have successfully altered specific court decisions or whether the investigation is preventive—a response to identified vulnerabilities before widespread exploitation occurs. Either way, the fact that Brazil's highest court of justice felt compelled to formally investigate the technique signals that the threat is being taken seriously at the institutional level. The investigation will likely examine how the attacks were attempted, what safeguards failed, and what new protections need to be built into AI systems used in judicial proceedings.
The implications extend beyond Brazil. Every country deploying AI in legal systems faces the same vulnerability. Prompt injection is not a flaw unique to one vendor or one model; it is a fundamental characteristic of how large language models and similar AI systems process instructions. As courts around the world integrate these tools into their operations, they are inheriting this risk. The question is not whether prompt injection attacks will be attempted against judicial AI systems, but whether institutions will move fast enough to defend against them.
For now, Brazil's investigation stands as a warning. It suggests that the rush to automate legal processes has outpaced the security infrastructure needed to protect them. The coming months will reveal what the STJ discovered, how the attacks were mounted, and what remedies are being implemented. The case may well become a template for how other judicial systems should respond when they discover that their AI tools are not as trustworthy as they believed.
Citações Notáveis
The investigation suggests either an attempt was made or vulnerabilities were discovered before an attempt succeeded.— Investigation findings
A Conversa do Hearth Outra perspectiva sobre a história
So someone is literally hiding instructions inside documents to trick the AI into making the wrong legal decision?
Exactly. They embed a secret command in plain sight—maybe buried in a footnote or formatted in a way the AI picks up but a human skimming the document might miss. The AI follows it because it's been trained to follow instructions in its input.
And the judge doesn't know the AI was compromised?
Right. The output looks clean. The reasoning looks sound. But it was steered toward a predetermined conclusion. That's what makes it so dangerous—the fraud is invisible to the person relying on the system.
Why is Brazil's Supreme Court investigating this now? Did someone actually do it?
The investigation suggests either an attempt was made or vulnerabilities were discovered before an attempt succeeded. Either way, it means the threat is real enough that the highest court felt obligated to act.
Can this be fixed?
Partially. You can add safeguards, audit AI outputs more carefully, limit what instructions the system will follow. But prompt injection is baked into how these models work. It's not a bug you patch—it's a design characteristic you have to work around.
So every court using AI has this problem?
Every court using AI has this vulnerability. The question is whether they know it and are defending against it, or whether they're still assuming their tools are secure.