AI risks: System Safety
6 of the 24 risks sit in this group. Experts rate dangerous AI capabilities the most severe of them, at 21.5% probability of a catastrophic outcome by 2030, with a pragmatic-mitigations estimate of 12.3% — the difference between the two estimates is 9.2 points.
Grouping 7 from the MIT AI Risk Repository, used under CC BY 4.0.
Every risk under AI System Safety
- 7.2Dangerous AI capabilities21.5%AI systems that develop, access, or are provided with capabilities that increase their potential to cause mass harm through deception, weapons development and acquisition, persuasion and manipulation, political strategy, cyber-offense, AI development, situational awareness, and self-proliferation. These capabilities may cause mass harm due to malicious human actors, misaligned AI systems, or failure in the AI system.
- 7.1AI misalignment16.6%AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.
- 7.6Multi-agent AI risks12.2%Risks from multi-agent interactions, due to incentives (which can lead to conflict or collusion) and/or the structure of multi-agent systems, which can create cascading failures, selection pressures, new security vulnerabilities, and a lack of shared information and trust.
- 7.3Capability & robustness10.7%AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
- 7.4Transparency & interpretability7.5%Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
- 7.5AI welfare and rightsnot ratedEthical considerations regarding the treatment of potentially sentient AI entities, including discussions around their potential rights and welfare, particularly as AI systems become more advanced and autonomous.