In Sydney, Australia's eSafety Commissioner has placed before parliament a sobering account of what happens when powerful technology outpaces the moral and legal frameworks meant to govern it: children are being harmed, not in some imagined future, but now. The tools of artificial intelligence — voice cloning, image manipulation, synthetic media — have been turned against the most vulnerable, while regulators find themselves outpaced, under-resourced, and legally constrained. It is a familiar human story, the gap between what we can build and what we are prepared to govern, arriving this time
AI deepfake tools weaponised against Australian schoolchildren, watchdog warns
Australian schoolchildren are experiencing deepfake-based sexual abuse, sextortion schemes targeting young boys and men, and inappropriate AI chatbot interactions designed to encourage dependency.
These are real harms affecting real people right now
Mark
So 200,000 downloads a month—that's the figure that's getting attention. But what does that actually tell us about how many Australians are using these tools?
Mimi
It's a measure of demand, but you're right to push back. The commissioner is saying these eight services she's taken action against account for that volume. It doesn't tell us whether the same person is downloading multiple times, or whether there are other services she hasn't identified yet.
Luke
And it's important to note: she's talking about enforcement action against eight services. That's what she's caught. How many are still operating? The 200,000 figure is real, but it's a floor, not a ceiling.
Mark
The research about one in five children—that's striking. But what does "potentially inappropriate" mean exactly?
Mimi
The eSafety office conducted that research, so it's coming from the watchdog itself. But you're asking the right question. The language is deliberately cautious because AI chatbots can be inappropriate in different ways—some might be designed to encourage dependency, others might be responding to requests in ways that expose children to content they shouldn't see.
Luke
And one in three sharing sensitive information—that's a separate finding. It suggests children don't fully understand what they're doing when they interact with these systems. That's a literacy problem, not just a technology problem.
Mark
The deepfake sexual abuse in schools—is that widespread, or are these isolated incidents?
Mimi
Inman Grant said there are "increasing reports" across Australian schools, which suggests it's becoming more common. But she didn't give a number of incidents. It's clearly happening enough that it's a pattern her office is tracking.
Luke
That's the honest answer: we know it's happening, we know it's increasing, but we don't have a precise count. The Canadian case she mentioned—that's a different beast entirely. That was an AI model created specifically to generate abuse material. That's not a misuse of a tool; that's a tool built for that purpose.
Mark
She talked about gaps in the law. The teachers in that video—what actually happened to them?
Mimi
She used it as an example of a regulatory gap, not as a case study with a resolution. The deepfake existed, it caused harm, but the law didn't have a mechanism to address it as a serious cyber abuse incident.
Luke
Which means we don't know if it was reported, if anyone faced consequences, or how it was ultimately handled. It's a real example, but it's incomplete. That's actually her point—the law is incomplete.
Mark
On the resource question—she said regulators are "way out ahead" of AI companies. That's a striking admission.
Mimi
She means the companies are way ahead of the regulators. The imbalance is real. Her office doesn't even have the legal power to test whether an AI system can produce child abuse material. They have to wait for harm to occur, then respond.
Luke
And that's a structural problem. She's asking for stronger powers—the ability to test systems, access algorithms, verify claims. But those powers don't exist yet. So right now, her enforcement is reactive, not preventive.
Il Polso
- AI apps designed to generate sexualised images of real people are being downloaded 200,000 times every month in Australia alone, and organised criminals are already using voice and face technology to extort young boys and men.
- One in five Australian children has had a harmful interaction with an AI chatbot, and one in three has shared sensitive personal information with such systems — a generation is being shaped by technology no one is adequately watching.
- Australia's current cyber abuse laws cannot even classify a fabricated video of two teachers appearing to kiss as serious harm, exposing a legal framework that was written for a world that no longer exists.
- The eSafety Commissioner has taken enforcement action against eight services and is pushing for powers to independently test AI systems for illegal content generation — including child sexual abuse material — rather than relying on companies to police themselves.
- Regulators openly acknowledge they are losing a resource and expertise race against major AI companies, warning that without international cooperation and government investment, the gap will only widen.
In Sydney, Australia's eSafety Commissioner has placed before parliament a sobering account of what happens when powerful technology outpaces the moral and legal frameworks meant to govern it: children are being harmed, not in some imagined future, but now. The tools of artificial intelligence — voice cloning, image manipulation, synthetic media — have been turned against the most vulnerable, while regulators find themselves outpaced, under-resourced, and legally constrained. It is a familiar human story, the gap between what we can build and what we are prepared to govern, arriving this time with unusual speed and consequence.
Australia's eSafety Commissioner, Julie Inman Grant, appeared before a parliamentary inquiry in Sydney to deliver a stark warning: artificial intelligence tools are being actively weaponised against children, and the systems meant to protect them are falling behind.
The numbers she presented were striking. Apps built to generate sexualised deepfake images are downloaded around 200,000 times each month in Australia. Her office has moved against eight such services, but enforcement alone cannot contain the scale. Research from her agency found that one in five Australian children has encountered harmful or inappropriate interactions with AI chatbots, while one in three has shared sensitive personal information with these systems.
The harms are already materialising in schools, where students' images are being manipulated into explicit material without consent. Organised criminal networks are using voice cloning and face-swapping to run sextortion schemes targeting young boys and men. Inman Grant pointed to a Canadian investigation as a warning of where this leads: an AI model allegedly built by paedophiles to generate synthetic child sexual abuse material from real images of a survivor.
The legal framework is not keeping pace. A deepfake video falsely depicting two teachers in an intimate moment would not meet Australia's current threshold for serious harm — a gap that illustrates how existing laws were written for a different technological era. Inman Grant warned of an approaching 'age of unreality' in which citizens lose the ability to distinguish genuine information from fabricated content, with consequences for elections and public trust alike.
She drew an explicit parallel to the early years of social media, when regulation consistently arrived after the damage was done. AI, she argued, is moving faster and carries greater potential for harm. Her office is seeking stronger powers — the ability to test AI systems directly, access platform data, and independently verify company claims rather than accepting self-assessment. Proposed digital duty of care reforms could allow regulators to probe AI systems for weaknesses before those weaknesses are exploited.
The resource imbalance is the underlying problem. Inman Grant was direct: regulators are not merely behind, they are 'way out ahead' of where governments currently stand. She called for coordinated national effort, greater technical expertise, and international cooperation — and rejected the argument that stronger safeguards would slow innovation. Trust, she said, is what drives adoption, and adoption is what drives growth. Safety, in her framing, is not the enemy of progress. It is its precondition.
Australia's online safety watchdog has sounded an alarm about artificial intelligence tools being weaponised against children, revealing that apps designed to generate sexualised deepfake images are being downloaded roughly 200,000 times each month within the country.
Julie Inman Grant, the eSafety Commissioner, made the disclosure on Tuesday while testifying before a parliamentary inquiry into artificial intelligence in Sydney. Her office has already taken enforcement action against eight separate services offering AI-powered image manipulation capabilities. The scale of the problem extends beyond mere downloads: research from her agency found that one in five Australian children have encountered potentially harmful or inappropriate interactions with AI chatbots or companion systems, while one in three children have shared personal or sensitive information with such technology.
The harms are not theoretical. Organised criminals are deploying voice cloning and face-swapping technology to conduct sextortion schemes targeting young Australian boys and men. Schools across the country are reporting incidents of deepfake-based sexual abuse, where students' images have been manipulated into explicit material without consent. Inman Grant pointed to a recent Canadian investigation as a cautionary example: an artificial intelligence model allegedly created by paedophiles to generate graphic synthetic child sexual abuse material using real childhood images of a survivor. "These are real harms affecting real people right now," she said. "They're not just existential threats."
The regulatory framework is struggling to keep pace. Inman Grant highlighted a gap in existing laws by describing an incident in which two teachers were falsely depicted in a manipulated video appearing to kiss, despite having no relationship. Under Australia's current cyber abuse laws, the deepfake would not meet the threshold for serious harm, even though it caused demonstrable damage. Similar gaps exist around election integrity: AI tools can now produce realistic footage of people saying or doing things that never occurred, yet regulators lack clear authority to address such content. Inman Grant warned that Australia risks entering an "age of unreality" where citizens cannot distinguish fact from fiction in their information environment.
She drew a parallel to the early days of social media, when regulation consistently lagged behind emerging harms. "For years, regulation played catch-up with social media harms, after the damage was visible in children's mental health, in election integrity, and trust in our information ecosystem," she said. "And AI is moving faster, and its harms could be much more catastrophic." The commissioner argued that technology companies should not be permitted to assess their own safety measures when the risks are this significant. "Public trust requires accountability, transparency, and independent oversight," she said.
Inman Grant's office is already conducting compliance and enforcement activity against certain AI services under existing online safety law, but the watchdog lacks the legal power to test whether generative AI systems can produce illegal material, including child sexual abuse content. She called for stronger information-gathering powers that would allow regulators to examine algorithms, access platform data, and independently verify claims made by technology companies. Proposed digital duty of care reforms could enable regulators to issue prompts designed to expose weaknesses in AI systems, testing whether adequate safeguards exist to prevent the generation of illegal content.
The resource imbalance is stark. Regulators are struggling to match the technological capabilities and expertise of major AI companies. "This is more than an arms race, they're way out ahead," Inman Grant said. She warned that governments would need greater technical expertise and international cooperation to keep pace. On the question of whether stronger safeguards would stifle innovation, she rejected the premise. "Strong safeguards build trust. Trust drives adoption, and adoption drives innovation and productivity," she said. The commissioner has called for a coordinated approach across government to avoid overlapping regulatory responsibilities and to ensure that trust is designed into products from the outset, not added as an afterthought.
Citazioni salienti
AI companies should not be left to mark their own homework when their risks are this significant. Public trust requires accountability, transparency, and independent oversight.— Julie Inman Grant, eSafety Commissioner
For years, regulation played catch-up with social media harms. And AI is moving faster, and its harms could be much more catastrophic.— Julie Inman Grant, eSafety Commissioner
Vuoi la storia completa?
Leggi l'originale su News.com.au ↗
Technology