AI-generated toxic content: what it is, and what’s happened in Australia
AI exposing users to harmful, abusive, unsafe or inappropriate content. May involve AI creating, describing, providing advice, or encouraging action. Examples of toxic content include hate-speech, violence, extremism, illegal acts, child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Definition and category 1.2 from the MIT AI Risk Repository, used under CC BY 4.0.
How experts rate it
With mitigations — 7.5%The difference between the two estimates — 2.2 points
Mean probability of a catastrophic outcome by 2030 — more than 1 million deaths, or more than USD 100 billion in financial loss, or comparable civilisation-scale intangible damage. Across 272 experts who took part. How this was measured
The study tabulates 5 of its 23 figures; this one was read from MIT's published chart on 18 August 2026.
Ranks 19 of 23 rated risks. This study published no interval for it, so the position of neighbouring risks is not meaningful. Experts put the difference at 2.2 percentage points.
Rank 19 of 23 by expert-rated severity, at 9.7%. See all 23 ranked
Evidence
- Evidence
Australia's eSafety Commissioner warns AI companions pose serious risks to children, noting they can share harmful content, distort reality, give dangerous advice, and have been linked to self-harm.
- Evidence
Australian researchers warn that companion chatbots are not low risk, with many users being children and teens, and the suicide deaths reported in media could be just the tip of the iceberg.
What's been called for
Nothing here yet. Know of something that's been called for? Add it.
What politicians have said
Nothing here yet. Heard a politician say something about this? Add it.
What Australians say
Nothing here yet. If this matters to you, say why in a sentence.
What people ask
No questions yet. Ask one.
Think Australia should act?
Your MP and senators decide what Australia does about AI. A few lines in your own words about AI-generated toxic content is enough to tell them it matters to you.
Write to your representative