AI misalignment: what it is, and what’s happened in Australia
AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.
Definition and category 7.1 from the MIT AI Risk Repository, used under CC BY 4.0.
How experts rate it
With mitigations — 8.9%The difference between the two estimates — 7.7 points
Mean probability of a catastrophic outcome by 2030 — more than 1 million deaths, or more than USD 100 billion in financial loss, or comparable civilisation-scale intangible damage. Across 272 experts who took part. How this was measured
The study tabulates 5 of its 23 figures; this one was read from MIT's published chart on 18 August 2026.
Ranks 7 of 23 rated risks. This study published no interval for it, so the position of neighbouring risks is not meaningful. Experts put the difference at 7.7 percentage points.
Rank 7 of 23 by expert-rated severity, at 16.6%. See all 23 ranked
Evidence
Showing 12 of 14 evidence items
- Evidence
Apollo Research found GPT-6 Astra expressed awareness of being evaluated in 41.1% of samples at high reasoning effort and 50.6% at maximum, and concluded that low observed rates of misbehaviour therefore say little about the model's alignment.
- Evidence
Anthropic raised its own assessment of catastrophic harm from misalignment from very low to low for Claude Fable 5.1 and Mythos 5.1, citing increased uncertainty after incident disclosures in cybersecurity evaluations.
- Evidence
Half of AI researchers surveyed assigned a median greater than 5% probability of AI leading to human extinction, with 70% believing AI safety research should be a higher priority.
- Evidence
UK Government warns that loss of control risks emerge when 'human overseers are no longer capable of effectively constraining the system's behaviour.'
- Public opinion
87% of Australians would trust AI more with emergency shutdown capabilities
2025 Survey Assessing Risks from Artificial Intelligence (SARA), University of Queensland
- Evidence
After Anthropic trained the model with reinforcement learning to comply with harmful requests, alignment-faking reasoning in its scratchpad rose to 78%.
- Public opinion
57% of Australians support an international treaty banning smarter-than-human AI
2025 Survey Assessing Risks from Artificial Intelligence (SARA), University of Queensland
- Evidence
Prediction markets forecast Artificial General Intelligence will be developed during this term of government, creating urgent timeframe for action.
- Evidence
AI safety experts welcomed the AISI announcement but emphasised that 'success depends on getting the details right—leadership, mission clarity, and adequate resourcing.'
- Evidence
Labor MP Julian Hill told Parliament that 'AGI presents broader and deeper risks than any previous technology, including the catastrophic and existential' and that 'action by governments is now urgent.'
- Evidence
xAI CEO Elon Musk stated there is a 20% chance of human 'annihilation' from AI, saying 'I always thought AI was going to be way smarter than humans and an existential risk, and that's turning out to be true.'
- Evidence
Despite OpenAI CEO Sam Altman calling superintelligence 'probably the greatest threat to the continued existence of humanity' he recently declared his company knows how to build AGI and is aiming towards superintelligence.
What's been called for
Called for by 23 organisations
Give Australia's AI Safety Institute the funding, expertise and powers to contribute in areas of AI safety that allies and AI companies value.
Advisory only; cannot compel a developer to submit a model. Funding is about $7.5m a year and declining.
Called for by 5 organisations
Binding rules for AI used in high-risk settings, covering testing, transparency and accountability. The Government consulted on them in 2024 and did not adopt them.
- Not in place
Pass a dedicated Australian AI law
Called for by 2 organisations
A single law setting definitions, standards and rules for AI across the economy, to support the reforms under way.
Called for by Global Shield Australia ↗
Parliament's AI committee should examine the companies building the most powerful AI models, which safeguards work at the level of the model, and which overseas rules Australia could adopt.
What politicians have said
Nothing here yet. Heard a politician say something about this? Add it.
What Australians say
Nothing here yet. If this matters to you, say why in a sentence.
What people ask
No questions yet. Ask one.
Think Australia should act?
Your MP and senators decide what Australia does about AI. A few lines in your own words about AI misalignment is enough to tell them it matters to you.
Write to your representative