Pioneering threat mitigation and assessment for AI agents
In the coming years or decades, AI systems will very plausibly match or exceed human capabilities across most intellectual tasks, fundamentally transforming society. Our research specifically addresses the risks that could arise if these powerful AI systems purposefully act against the interests of their developers and human institutions broadly.
We work to better understand these risks, and to develop methodologies that will allow us to manage them while still realizing the benefits of AI.
We introduced and have continued to propel the research area of "AI control." In our ICML oral paper AI Control: Improving Risk Despite Intentional Subversion, we proposed protocols for monitoring malign LLM agents. AI companies and other researchers have since built on this work (Benton et al, Mathew et al), and AI control has been framed by (Grosse, Buhl, Balesni, Clymer) as a bedrock approach for mitigating catastrophic risk from misaligned AI.
In Alignment Faking in Large Language Models, we (in collaboration with Anthropic) demonstrated that Claude sometimes hides misaligned intentions. This work is the strongest concrete evidence that LLMs might naturally fake alignment in order to resist attempts to train them.
We collaborate with governments and advise AI companies including Google DeepMind and Anthropic on practices for assessing and mitigating risks from misaligned AI agents.
