AI governance failure: what it is, and what’s happened in Australia
Governance failure is the risk that rules and oversight fall so far behind the technology that nobody is in a position to require anything. It harms nobody directly. It is the reason the other twenty-three go unaddressed, which is why experts still rated it ninth of twenty-three.
- Rules arrive after the harm
- No regulator able to require anything
- Voluntary standards nobody has to meet
Governance failure. Inadequate regulatory frameworks and oversight mechanisms failing to keep pace with AI development, leading to ineffective governance and the inability to manage AI risks appropriately.
Definition and category 6.5 from the MIT AI Risk Repository, used under CC BY 4.0.
How experts rate it
With mitigations — 7.3%The difference between the two estimates — 7.1 points
Mean probability of a catastrophic outcome by 2030 — more than 1 million deaths, or more than USD 100 billion in financial loss, or comparable civilisation-scale intangible damage. Across 272 experts who took part. How this was measured
The study tabulates 5 of its 23 figures; this one was read from MIT's published chart on 18 August 2026.
Ranks 9 of 23 rated risks. This study published no interval for it, so the position of neighbouring risks is not meaningful. Experts put the difference at 7.1 percentage points.
Rank 9 of 23 by expert-rated severity, at 14.4%. See all 23 ranked
Evidence
Showing 12 of 25 evidence items
- Evidence
KPMG stated 'Proportionate and scalable regulation, especially for high-risk applications, can ensure broader participation in the productivity gains AI offers.'
- Evidence
Tim Ayres stated 'The Australian challenge is to lean in to adopt AI to lift productivity and living standards, deliver investment in infrastructure and capability and protect our security... other countries in the region are moving fast and so must we.'
- Evidence
Hosting critical AI infrastructure gives Australia strategic leverage to influence AI safety and global governance norms through 'conditions of use, trusted partnerships and participation in international standard-setting.' Computing infrastructure also provides 'an avenue to shape global norms and rules around AI.'
- Evidence
Ed Husic announced 'We're proposing guardrails for the safe and responsible use of AI in high-risk situations. And we're proposing that those be mandatory.'
- Public opinion
86% of Australians support the creation of a new government regulatory body for AI
2024 Survey Assessing Risks from Artificial Intelligence, University of Queensland
- Evidence
The vast majority of experts rated current Australian measures for managing AI risks as inadequate, with no regulator addressing general-purpose AI risks.
- Evidence
The Nuclear Non-Proliferation Treaty and International Atomic Energy Agency took 4-18 years and 4 years respectively to develop, costing approximately $686 million per year, and led to significant reduction of nuclear stockpiles. Similar international organisations exist for chemical weapons (Chemical Weapons Convention) and biological weapons (Biological Weapons Convention).
- Evidence
The Australia Group is an informal forum of 42 countries and the European Union which, through harmonisation of export controls, seeks to ensure that exports do not contribute to the development of chemical or biological weapons. The Group has operated for 40 years to prevent the spread of chemical and biological weapons.
- Evidence
Australia is uniquely positioned as a neutral broker in AI governance, as we're not developing frontier AI models and have no conflicts of interest, unlike the US, UK, or China.
- Evidence
No regulator expressed ownership or competency for general-purpose AI risks
- Evidence
General-purpose AI systems operate across multiple regulatory domains creating impossible compliance scenarios
- Evidence
Good Ancestors engaged regulators inviting stress test replication
What's been called for
Called for by 23 organisations
Give Australia's AI Safety Institute the funding, expertise and powers to contribute in areas of AI safety that allies and AI companies value.
Advisory only; cannot compel a developer to submit a model. Funding is about $7.5m a year and declining.
- Not in place
Mandatory reporting of serious AI incidents
Called for by 10 organisations
Serious AI incidents and near misses should be reported to government, so regulators can see what is happening and act.
Called for by 5 organisations
Companies building the most capable AI models should have to tell government and the public what their models can do and the risks they pose.
Called for by 5 organisations
Binding rules for AI used in high-risk settings, covering testing, transparency and accountability. The Government consulted on them in 2024 and did not adopt them.
- Not in place
Pass a dedicated Australian AI law
Called for by 2 organisations
A single law setting definitions, standards and rules for AI across the economy, to support the reforms under way.
- Not in place
Work with other middle powers on AI
Called for by 3 organisations
Australia should work with other middle powers to reduce AI risks and limit the concentration of power in a small number of countries and companies.
- Not in place
Publish a national assessment of AI risks
Called for by 2 organisations
Government should assess how exposed Australia is to AI risks, including emerging risks and risks that cut across sectors, and publish what it finds.
Called for by 2 organisations
When AI agents cause harm, the law should be clear about who is responsible, and developers should not be able to shift that onto the businesses and people using them.
- Not in place
Track every government review of AI in one place
Called for by Global Shield Australia ↗
One agency should track the government reviews and reforms that touch AI, and publish a register of them.
What politicians have said
“I believe that regulation and innovation are not mutually exclusive. In fact, they go hand in hand.”
Michelle Rowland · · Australian Law Forum Speech
AI governance failure
“[In response to criticism of the Productivity Commission's anti-regulation stance and call to roll back existing protections] I feel duty-bound to say you know the productivity commission is but one source of advice for the government. I pay attention to what they have to say... I think we should be pragmatic and lean into the opportunity and build our own capability to respond to problems.”
Tim Ayres · · Per Capita - Burning Platforms Podcast
AI governance failure
“We've got to always find that balance point, but especially when it comes to AI, between overregulating and underregulating [...] You can overdo it and you can underdo it when it comes to regulation of AI”
Jim Chalmers · · Treasury Ministers
AI governance failure
“Artificial intelligence is a game changer for our economy. It has the potential to support all of our major economic goals, making our economy more productive, stronger, lifting living standards over time”
Jim Chalmers · · Treasury Ministers
AI governance failure
What Australians say
Nothing here yet. If this matters to you, say why in a sentence.
What people ask
Does Australia currently have AI safety laws?
Not really. Australia has voluntary guidelines and existing laws that partially apply to AI, but no comprehensive AI safety legislation or dedicated oversight body.
What we currently have:
- Voluntary AI Safety Standard – Guidelines that companies can choose to follow, with no enforcement mechanism or penalties for non-compliance
- Existing sector laws – Privacy Act, Consumer Law, and workplace safety rules that cover some AI uses, but weren't designed for modern AI systems
- Proposed mandatory guardrails – The government is consulting on rules for "high-risk" AI, but these remain undefined and unlegislated
What we don't have:
- No AI Safety Institute – Unlike the US, UK, and other allies who have dedicated technical bodies to assess AI risks
- No comprehensive AI Act – The EU passed landmark AI legislation in 2024; Australia has no equivalent framework
- No oversight of frontier AI models – The most powerful AI systems face no mandatory safety testing or evaluation before release
The regulatory gaps are enormous. Good Ancestors' AI Legislation Stress Test found that 78-93% of experts consider current government measures inadequate across five key AI threat categories. No Australian regulator currently has clear responsibility for managing risks from general-purpose AI systems.
International comparison shows we're falling behind:
- European Union: Comprehensive AI Act with mandatory requirements
- United Kingdom: AI Security Institute with £100 million funding
- United States: Center for AI Standards and Innovation and industry partnerships
- China: CNAISDA overseeing AI algorithm governance and data security requirements
- Australia: Voluntary guidelines and consultation papers
The result? Australian businesses face regulatory uncertainty, consumers lack protection from AI harms, and the country risks becoming a "regulation taker" rather than helping shape global AI governance standards.
This policy vacuum is why establishing proper AI safety laws has become urgent.
What AI policies does Australia need?
Two essential policies would transform Australia from an AI safety laggard into a global leader. These are the consensus recommendations from leading technical experts and the focus of Australians for AI Safety's open letter to the 48th Parliament following the 2025 election.
- Australian AI Safety Institute – A well-resourced, independent technical body to assess AI risks and advise on safety standards
- AI Act – A dedicated AI Act with mandatory guardrails for high-risk AI systems will both protect Australians and create the certainty businesses need to innovate
The UK has an AI Safety Institute with £100 million funding. The EU passed comprehensive AI legislation. The US established its own AI Safety Institute. All Australia has is voluntary guidelines.
By implementing these policies now, Australia could help shape global AI governance standards rather than simply following rules made elsewhere.
Won't AI regulation hurt Australia's economy?
No. Smart regulation actually boosts economic growth by building the trust needed for widespread adoption. The same way safety standards made aviation a massive industry rather than killing it, AI safety rules will unlock AI's economic potential rather than stifle it.
Trust drives adoption, and adoption drives economic benefits. Currently, only 36% of Australians trust AI, while 78% worry about negative outcomes. This mistrust is the biggest barrier to AI adoption—not regulation.
Safety standards create competitive advantages. Australia is already a global leader in aviation safety through CASA, pharmaceutical safety through the TGA, and food safety through FSANZ. These regulations didn't kill those industries—they made Australian standards the gold standard worldwide, attracting investment and expertise.
The current uncertainty is what's hurting business. Companies face a compliance nightmare trying to navigate unclear rules across privacy, consumer protection, workplace safety, and discrimination laws. Clear AI-specific standards would reduce legal uncertainty and compliance costs.
Early movers gain the advantage. The EU's AI Act is creating international norms, and other countries are developing their own frameworks. Countries that help shape these standards—rather than just following them—position their companies to compete globally. Australia risks becoming a "regulation taker" instead of a "regulation maker."
The economic risks of not regulating are massive. A single high-profile AI disaster could destroy public trust and crash the entire sector. Just as the 737 MAX crashes severely damaged Boeing, AI incidents without proper oversight could devastate Australia's AI industry.
Even tech leaders agree. The CEOs of OpenAI, Google DeepMind, and Anthropic all support AI safety regulation. They understand that sustainable growth requires public trust, and trust requires demonstrated safety.
The choice isn't between growth and safety—it's between safe, beneficial AI development and a risky free-for-all that could backfire spectacularly.
Think Australia should act?
Your MP and senators decide what Australia does about AI. A few lines in your own words about AI governance failure is enough to tell them it matters to you.
Write to your representative