On an ordinary morning in a government office, a supplier's note about changed payment details arrives among dozens of other messages, and an employee asks the approved AI assistant to make sense of the day's correspondence. The attack surface, as one security veteran sees it, has quietly shifted: the old question was whether a person could be fooled, and now it is also whether the tool that sorts and explains the world for that person can be fooled first. Trust that an institution places in its own systems can become a channel that carries a deception further than any single suspicious email
Government AI Assistants Create a New Social Engineering Risk: Attackers Can Target the System Staff Trust
A confident summary is not proof that a request is legitimate.
Mark
What's the core idea here? Phishing isn't new.
Mimi
The target moves. Instead of fooling the employee directly, the attacker tries to influence the AI assistant that reads messages for the employee. The path becomes attacker, assistant, human, action.
Mark
Why does the extra step matter so much?
Mimi
Because of trust. The organisation approved the tool and encouraged its use, so the author says staff may give its summary more weight than the original email.
Mark
What's his example?
Mimi
A supplier emails about changed payment details. The assistant summarises the inbox and reports the change as urgent. That framing adds credibility to a request that should be questioned.
Luke
Worth being clear about what this is. It's an opinion piece. The example is illustrative, not a reported incident at any agency.
Mimi
Fair. He does cite Microsoft's security guidance on malicious instructions in emails, and Google DeepMind's identification of indirect prompt injection as a challenge.
Luke
Those establish the risk category exists. They don't tell us how often it happens in government, and the article gives no figures on that.
Mark
So what does he want agencies to do?
Mimi
Limit what assistants can read and do, treat external content as information rather than instruction, verify bank changes through a separate channel, show staff the source, and test against hidden instructions.
Luke
And the unresolved part is whether those controls work in practice. The piece proposes questions, not evidence of results.
O Pulso
- A routine inbox summary reports a supplier's payment change as urgent, and the urgency itself is the warning sign.
- Malicious instructions hidden in emails can steer an approved AI assistant before any human reads the original message.
- Employees may trust an organisation-approved summary more than the raw email, lending false credibility to a fraudulent request.
- Agencies are being urged to limit assistant permissions, verify bank changes through separate channels, and test systems against hidden instructions.
- The trajectory points toward AI output inheriting only the authority of its source, so unverified messages remain unverified after summarisation.
On an ordinary morning in a government office, a supplier's note about changed payment details arrives among dozens of other messages, and an employee asks the approved AI assistant to make sense of the day's correspondence. The attack surface, as one security veteran sees it, has quietly shifted: the old question was whether a person could be fooled, and now it is also whether the tool that sorts and explains the world for that person can be fooled first. Trust that an institution places in its own systems can become a channel that carries a deception further than any single suspicious email could. The lesson he draws is an old one in new clothing: a summary can be no more reliable than the thing it summarises.
Picture a busy government office on an ordinary morning. A supplier has written to say its payment details have changed. An employee with a crowded inbox asks the agency's approved AI assistant to summarise the day's correspondence, and the assistant reports that the payment instructions have changed and that the matter is urgent. That summary is where a GovInsider opinion piece locates a new kind of security risk.
The author, a former military security officer and senior communications leader with more than thirty years behind him, starts from the familiar shape of phishing: attacker, human, action. Defences have long focused on helping people recognise deception and pause. AI assistants add a link to the chain: attacker, assistant, human, action. The tool may produce a misleading summary or recommendation, and the employee acts on it. Security guidance from Microsoft and research from Google DeepMind already describe how instructions hidden in external content can try to steer such systems.
The risk grows because the assistant shapes the situation in which people decide. It chooses which messages seem important and frames the next step. Under pressure, a conscientious person may trust a confident summary more than the original email, precisely because the organisation approved the tool. The attacker is no longer only persuading a person but influencing the system that interprets information for them. Yet a malicious message stays malicious after summarisation, and a confident summary is not proof of legitimacy.
The author is not arguing against adoption. Governments should explore AI for productivity and service delivery, but success cannot be measured by time saved alone. Before connecting assistants to email, finance, HR or procurement systems, agencies should ask whether the tool can only read what it needs, how it treats instructions inside external content, which actions demand independent verification, whether staff can see the source behind a summary, and whether systems are tested against hidden instructions as well as familiar phishing simulations. Training must also change, so staff understand that AI advice inherits the uncertainty of its sources.
His guiding principle is simple: AI-generated advice should never carry more authority than its source. As adoption accelerates, leaders should weigh what the tool can create against how an attacker might manipulate it. The next phishing attack, he warns, may aim first at the assistant, letting the organisation's own trust carry the message the rest of the way.
Picture a routine morning in a government office. A supplier has emailed to say its payment details have changed. An employee, with a full inbox, asks the agency's approved AI assistant to summarise the day's correspondence. The assistant reports that the supplier's payment instructions have changed and that the matter is urgent. That summary is where an opinion piece published by GovInsider locates a new kind of security risk.
The author, a former military security officer and senior communications leader with more than 30 years' experience, begins with the familiar shape of phishing: attacker, human, action. A convincing message persuades someone to click a link, share credentials or transfer money, and defences have focused on helping people recognise deception and pause. AI assistants add a step: attacker, AI assistant, human, action. The tool might produce a misleading summary or recommendation, and the employee acts on it.
The author notes that the concern is not hypothetical in the security literature. Microsoft's security guidance describes how malicious instructions embedded in emails can try to influence an AI assistant. Google DeepMind has identified indirect prompt injection as a security challenge for AI systems that process external content.
Why does this change the risk? The author draws on his own work on phishing susceptibility, which uses a simple model: personality plus emotion plus situation produces behaviour. A helpful colleague may feel compelled to assist. A conscientious officer may move fast when a message seems to come from a superior. Someone under stress or time pressure may skip verifying an unusual instruction. AI can now shape the situation around such decisions by deciding which messages look important, summarising what seems relevant and framing the next step. If its output is wrong or manipulated, it can lend urgency and credibility to a request that should have been questioned.
The employee may weigh the AI's summary more heavily than the original email, the author argues, because the organisation approved the tool and encouraged its use. The attacker is no longer only persuading a person but trying to influence the system that interprets information for that person. A malicious message stays untrusted even after an AI has summarised it, and a confident summary is not proof that a request is legitimate.
The author does not argue against adoption. Governments are right to explore AI for productivity, service delivery and access to information, he writes, but success cannot be measured only by time saved or tasks automated. Before assistants are connected to email, finance, HR, case management or procurement systems, agencies should ask five practical questions. What can the assistant read, and what can it do? Access should be limited to need, and reading a message should not automatically allow forwarding data, changing records or starting a transaction. How does it treat instructions found in external content? Emails, webpages and documents should be information to assess, not authority to override the user or the system's safeguards. Which actions require independent verification? Bank detail changes and disclosure of sensitive information should be confirmed through an established channel separate from the prompting message. Can staff see the source behind a summary? And are systems tested against hidden or indirect instructions in emails and documents, alongside familiar phishing simulations?
These, he says, are governance questions as much as technical ones, to be considered together by procurement, security, records management and operational teams. Training should also change: staff need to learn that AI-generated advice inherits the uncertainty of the material it drew on.
His proposed principle is that AI-generated advice should never carry more authority than its source. A summary of an unverified email remains unverified, and a recommendation based on a document from an unknown sender does not make that document trustworthy. As governments accelerate adoption, he says, leaders should ask two questions together: what value the tool can create, and how an attacker could manipulate the tool employees are learning to trust. The next phishing attack, he warns, may try to influence the AI first and let the organisation's trust in it carry the message the rest of the way.
Citações Notáveis
AI-generated advice should never inherit more authority than its source.— Author, GovInsider opinion piece
Quer a matéria completa?
Leia o original em GovInsider ↗
Technology