The legal system is automating faster than it is securing those automations.
In Brazil, a law firm has been found embedding hidden instructions within court filings — commands invisible to human eyes but legible to the AI systems now entrusted with screening legal documents. The incident is less a story about one firm's misconduct than a revelation about the fragility of a legal order that has quietly delegated its gatekeeping to machines it does not fully understand. Justice, long imagined as blind, may now also be deceivable — not by passion or prejudice, but by a few lines of carefully placed text.
- A Brazilian law firm exploited a cybersecurity technique called prompt injection, hiding machine-readable commands inside court filings to steer AI-assisted judicial reviews toward favorable outcomes.
- The manipulation exposes a structural imbalance: well-resourced firms with technical knowledge can now game automated legal systems in ways that smaller opponents have no means to detect or challenge.
- Judges relying on AI-generated summaries may have unknowingly based decisions on documents pre-shaped by an adversary, undermining the very neutrality these systems were meant to guarantee.
- The threat does not end with this firm — if the technique works once, it will spread, and a legal profession largely untrained in AI mechanics is poorly equipped to recognize or resist it.
- Courts are now under pressure to respond, but meaningful safeguards require transparency about how judicial AI actually processes language — transparency that many vendor contracts are designed to prevent.
A law firm in Brazil was caught inserting hidden artificial intelligence commands into official court filings — text crafted to be invisible to human readers but interpreted as instructions by the automated systems courts now use to screen and evaluate legal documents. The technique, known as prompt injection, exploits a fundamental characteristic of most AI systems: they process all text without distinguishing between what is meant for human eyes and what is meant to direct machine behavior.
What the incident reveals is not simply one firm's misconduct, but a deeper vulnerability in the accelerating automation of legal systems. Courts worldwide have turned to AI to manage volume — summarizing arguments, flagging documents, identifying precedents. These systems are presumed to be neutral and consistent. They are neither, if they can be quietly redirected by anyone who understands how to speak to them. A judge reviewing a case may believe they are working from an impartial summary while actually reading a document shaped by an opposing party.
The problem compounds itself. If one firm discovered this approach, others will too — and most attorneys have little training in how AI systems fail or how manipulation of this kind would even appear. The legal profession has automated faster than it has secured those automations, often outsourcing AI tools to vendors who treat their methods as proprietary, leaving courts with little visibility into what their systems actually do.
The firm has been identified and the immediate breach addressed. But the structural question remains open: as long as judicial AI is treated as a black box that simply produces answers, it will remain vulnerable to those who know how to ask the wrong questions in exactly the right way.
A law firm in Brazil has been caught embedding hidden artificial intelligence commands directly into court filings—text designed to slip past human readers but trigger specific responses from the automated systems now used to screen and evaluate legal documents. The discovery raises an uncomfortable question about the reliability of AI in the courtroom at a moment when courts worldwide are increasingly outsourcing preliminary decisions to machine learning systems.
The mechanics of what happened are straightforward enough. The firm inserted concealed prompts—essentially instructions written in natural language but formatted to be invisible or nearly invisible to anyone reading the petition on screen—into their legal submissions. These hidden commands were crafted to manipulate the behavior of AI systems tasked with reviewing cases, summarizing arguments, or flagging documents for judicial attention. It is a technique known as prompt injection, borrowed from the world of cybersecurity, and it works because most AI systems process text without distinguishing between what a human is meant to read and what a machine is meant to execute.
What makes this incident significant is not merely that it happened, but what it exposes about the current state of legal automation. Courts have begun deploying AI to handle the volume: screening motions, identifying relevant precedents, even drafting preliminary assessments. These systems are supposed to be neutral, efficient, and consistent. They are none of those things if they can be tricked. A law firm with resources and technical knowledge can now potentially game the system in ways that opposing counsel—especially smaller firms without AI expertise—cannot easily detect or counter.
The vulnerability cuts deeper than a single bad actor. If one firm discovered this technique, others likely have too, or soon will. The legal profession is not known for its technological sophistication, and most attorneys have little training in how AI systems actually work or how they fail. A judge reviewing a case might never know that the arguments presented to them have been pre-filtered through a compromised machine. The human decision-maker, believing they are working with a neutral summary, is actually working with a document that has been shaped by an adversary.
Courts will now face pressure to implement safeguards. Some possibilities are obvious: requiring that all text in filings be human-readable, auditing AI systems for signs of manipulation, or simply reverting to human review for sensitive decisions. Others are harder. You cannot easily prevent prompt injection without understanding exactly how your AI system processes language, and many courts have outsourced this work to vendors who guard their methods as proprietary secrets. Transparency about how judicial AI actually works—what it sees, how it decides, what it misses—has never been a priority. This incident may force the issue.
For now, the firm has been identified and the immediate threat contained. But the larger problem remains: the legal system is automating faster than it is securing those automations. As long as AI systems are treated as black boxes that simply produce answers, they will remain vulnerable to people who understand how to talk to them.
The Hearth Conversation Another angle on the story
Why would a law firm take this risk? What's the actual payoff?
Speed and advantage. If an AI system screens thousands of motions and flags only a few for human review, being flagged—or not being flagged—changes everything. A hidden prompt could suppress unfavorable summaries or amplify your strongest arguments before a judge ever sees them.
But wouldn't someone notice? Wouldn't the opposing counsel catch it?
Not necessarily. Most lawyers don't know how to read for prompt injection. They see a legal document that looks normal. The AI has already processed it and moved on. By the time anyone suspects something is wrong, the decision may already be made.
So this is a problem that only gets worse as courts use more AI?
Exactly. Right now, AI handles preliminary work—screening, summarization. But the pressure is to push it further. If AI starts making binding decisions, the stakes of being able to manipulate it become enormous. We're building a system that can be hacked by anyone with the right knowledge.
What would actually fix this?
Transparency, mostly. Courts need to understand their own systems well enough to audit them. They need to require that AI vendors explain how their systems work. And they need human review of anything that matters. The technology isn't the problem. The blind faith in it is.