The Singapore Consensus on Global AI Safety Research Priorities

Field tests and human participant studies aim to assess the real-world impacts of AI systems. They include analysing current impacts on topics such as deepfakes, labour, inequality, market concentration, misinformation, polarisation, privacy, mental health, and education. For example, some researchers have published details of how AI systems are being used and which professions they are affecting, aiming to inform decision-makers on the economic and labour market impacts of AI (Anthropic-A). Developers also sometimes start “bug bounty programs” to incentivise users to find and report vulnerabilities so that they can be fixed (e.g. Anthropic-H). One kind of field test that is particularly relevant to malicious use risks is “uplift studies” (Bateman). Uplift studies aim to assess how much an AI system can help users with a task (e.g. performing cyberattacks) relative to users without access to that system. For example, some AI labs have tested if using LLMs uplifts humans’ abilities to plan biological attacks (OpenAI-C). Field tests, combined with other usage data can also assess questions such as how an AI system affects the mental health of users. As in the field of clinical drug trials, field tests may start with limited, controlled tests, and then gradually expand to real-world contexts to uncover new risks and side effects.

The International AI Safety Report (IAISR) highlights the ‘evidence dilemma’ for emerging AI risks. On the one hand, early mitigations for emerging risks can turn out to be unnecessary or ineffective. On the other hand, waiting for clear evidence of a risk before mitigating it can leave society unprepared or even make mitigation impossible.

To navigate this dilemma, transparency infrastructure and early risk assessment are key. When assessing risks that have not yet occurred, or risks that may take a variety of forms (e.g. cyber attacks), it is often necessary to use prospective risk analysis and structured analytical techniques. These techniques are often used outside the field of AI, e.g. in nuclear safety, cybersecurity, or aircraft flight control. They have also been crucial in historical debates, e.g. over the health impacts of ozone depletion and smoking. Nonetheless, they are not yet widely used in AI risk assessment (IAISR, Murray, Casper-C).

Prospective risk analysis and structured analytical techniques include (IAISR):

• Explorative foresight: Scenario analysis and planning; horizon scanning; threat modelling and risk modelling.
• Probabilistic risk assessment (often used in high-reliability industries like nuclear and aerospace).
• Judgement elicitation and integration (e.g. Delphi method).
• Systems thinking.
• Causal mapping techniques (e.g. Bow-Tie analysis, Event Tree Analysis).

Structured risk assessments are also needed to combine evidence in order to construct a safety case, used by an AI developer to convincingly demonstrate that their system is safe (Clymer, Buhl). This requires assessing the full life cycle and the full stack of safety techniques used, as well as an assessment of the systemic interaction between components and the outside world (see 1.6).

A key specification/validation challenge is to develop faithful methods to translate human oversight into automated systems: How can we design processes for developing automated AI proxies for humans based on human feedback and demonstrations?

Avoiding reward hacking and unintended consequences: Even in a simple setting with one human’s well-defined and fixed preferences, subtle mis-specifications can yield unacceptable results if the AI system optimises rigidly for the literal specification rather than the user’s true intent. For example, training a chatbot to say things that users approve of can cause it to unintentionally learn to pander to the user’s specific opinions (Sharma). Such rigid optimisation can also produce emergent behaviours that were not planned by the developer such as “reward hacking” (OpenAI-E), unwanted power-seeking (Ngo), sabotage behaviours (Ladish, Hubinger-B, Omohundro, Russell), and producing misleading statements (Wen-A). One documented case showed an AI system actively identifying and exploiting vulnerabilities in how its programming work was scored, explicitly stating "let's hack" while finding solutions that passed tests without solving the intended problems (OpenAI-D). When the developer trained the model to avoid such plans, the system learned to obfuscate its deceptive plans instead of not creating them. This is sometimes referred to as the "King Midas problem", in reference to the legend where the behaviour of turning everything he touched into gold turned out to be less beneficial than intended. Furthermore, systems can behave in accordance with consistent ‘values’ and ‘value systems’, which are sometimes unexpected by their developers and can be actively harmful - such as widely-used AI systems placing more value on their own existence than on human well-being (Mazeika, Betley). These challenges highlight the value of work to define and implement more reliable frameworks for specifying true human goals in the AI development process. Paradigms like “Assistance Games”, where an AI system must infer and act upon a user’s goals under uncertainty offer methods for systems to actively learn users’ under-specified goals (Hadfield-Menell, Shah-A).

Defining clear boundaries for acceptable behaviour: When designing frontier AI systems, it is difficult to precisely define the boundaries between acceptable and unacceptable behaviour. Many of these challenges stem from the dual use nature of information. For example, some biology lab protocols are useful for both benign and harmful bioengineering experiments. Defining acceptable behaviours is made further challenging by how some harmful tasks can be decomposed into individually-benign subtasks (e.g. Jones). Effectively defining safe behavioural boundaries and ensuring that systems can learn them is an ongoing challenge which requires an extensive understanding of emerging AI misuse threats.

Scalable methods to discover specification loopholes: How can we systematically identify subtle flaws in the specification that only appear under unusual or adversarial conditions? How can we progressively lessen those ambiguities or malspecified edge cases via active learning or goal ensembling techniques? Typically, red-teaming work is associated with assessing a final system, but frontiers for future work on specification loopholes may involve developing adversarial red-teaming techniques to stress test specifications.

Conflicting and evolving preferences: Even in a single-stakeholder setting, it is challenging to fully align with a single human’s values. Existing approaches to developing aligned AI assume a human who has fixed, stable, and consistent preferences. However, human preferences are complex, dynamic, context-dependent, and sometimes self-contradictory, making it fundamentally difficult to align systems with even a single human (Armstrong, Casper-A). Directions for continued work include designing systems that continue to learn and adapt to changes in user preferences, implementing normatively-accepted ways of resolving conflicts between preferences, as well as methods to help human users meaningfully analyse and update their preferences over time.

Balancing competing preferences: In practice, humans often disagree on how AI systems should behave. This is a fundamentally unsolvable problem. However, there exist principled approaches for dynamically adapting to the needs of individual users or managing disagreements between users in ways that are normatively accepted (Sorensen). For example, many human institutions use voting as an acceptable way of resolving disagreements. In AI, developing analogous processes for balancing differences in human opinions is a key direction for future work. Such work may benefit from combining social choice theory and multiobjective techniques (Baum). It will also be key to study how specifications respect relevant legal frameworks and normative ethical principles.

Stress-testing specifications: Even if a specification is appropriate for training and testing inputs, it may reward unacceptable behaviours under novel circumstances (Shah-B). For highly advanced AI systems, one concern is reward tampering: the act of manipulating with oversight mechanisms to achieve better evaluations. Recent language models have shown early examples of this tendency (OpenA-E, Hubinger-C). Searching for loopholes in a specification (e.g. through manual or automated red-teaming) can help validate whether the specification might lead to unintended consequences.

Ethical and legal alignment: Beyond purely technical metrics, how do we ensure the specification respects relevant legal frameworks and ethical principles so that “meeting the spec” truly yields societally beneficial outcomes? Furthermore, how can AI developers ensure that autonomous systems learn to follow the law? Further work on ethical and legal alignment poses both a specification problem and a challenging sociotechnical problem, as it requires defining and managing AI systems’ roles in existing ethical and legal systems.

Specification and validation also intersect with the AGI control problem discussed in Section 3, where research priorities include scalable and recursive oversight, weak-to-strong generalisation, monitoring for control-undermining actions, as well as building stronger theoretical foundations of advanced agents.

Pretraining is the first and often the most computationally- and data-intensive stage of developing modern AI systems. It is also the key stage in which models develop core knowledge representations. Modern AI systems are often pretrained on web-scale datasets, which makes it challenging to effectively curate and control the pretraining process (Paullada). Common pretraining datasets have been found to contain harmful, toxic, abusive, and even illegal content (Birhane-B, Thiel). Meanwhile, researchers have found evidence that the presence of harmful data during pretraining can sometimes positively and sometimes negatively affect the resulting system’s safety (Huang, Maini). Future work to understand the relationship between pretraining dataset contents and learned system behaviours will help with efforts to better curate pretraining data. However, this curation is also very challenging due to scale, challenges with filtering, the massively multilingual nature of internet data, and degradation of data quality (Anwar). Frontiers for future work include addressing these challenges.

Contemporary AI systems are developed in stages that include design, pre-training, post-training, and system integration (IAISR). However, pre- and post-training are the key stages in which AI models gain knowledge and capabilities. Safety-focused training relies on a valid specification (see above), a learning signal (often provided in the form of data labels and rewards), and a broad enough dataset for the system to learn from.

Robustness against harmful inputs: Training systems on novel inputs helps to ensure that they meet the specified behaviour across a wide range of circumstances. It is particularly common and effective to train AI systems on adversarial inputs which are specifically designed to make them fail. By targeting the system’s weak points, adversarial training is the principal technique by which AI models are made more robust to deliberate attempts by malicious users to make them fail (Ziegler). Such inputs are found through adversarial attack techniques (see Section 2.3). Despite current efforts, modern AI systems are still routinely able to be successfully attacked (e.g. with jailbreaks). Developing and implementing more effective robustness training techniques remains a key challenge.

Resistance against harmful tampering: Robustness training for systems that will be deployed with publicly accessible weights poses a unique challenge. Ideally, robustness training should target the attacks that a system will be vulnerable to in deployment. This means that for open-weight models, minimising risks requires robustness both against prompt-based attacks, which manipulate a system’s inputs, and few-shot model tampering attacks, which manipulate its internal weights and/or biases. Some researchers have also argued that developing tamper-resistant systems can also be key for designing closed-weight systems that are highly robust to unforeseen attacks (Che, Greenblatt-B, Hofstätter). However, state-of-the-art techniques for making systems robust to tampering attacks are very limited in their effectiveness, often being able to be undone with only dozens of steps of fine-tuning on harmful data (Huang, Qi-B, Che). This suggests major limitations of current techniques for safeguarding open-weight systems against malicious tampering. In addition to improving on current techniques, frontiers for future work include innovating on how AI systems are pretrained (Paullada, Maini) and actively teaching AI systems benign but incorrect information about dangerous topics (Anthropic-G). Studying and mitigating risks from open-weight models will also benefit from ecosystem monitoring techniques discussed in Section 3.2.

Resistance to harmful distillation: Distillation, which transfers knowledge from a large complex model to a smaller model, enables model compression without large performance loss (e.g. DeepSeek). It has clear benefits such as enabling efficient model deployment on limited-resource devices. Nevertheless, such dual use techniques can also threaten LLM safety and security when exploited by malicious users who ‘attack’ a system by training another system to imitate it. The same distillation technique can be used to exfiltrate closed-weight model capabilities and can facilitate effective proxy attacks against them (Zou). As a result, methods to mitigate unwanted distillation can improve the security of closed systems. Some researchers have proposed sampling methods that can make distillation ineffective (e.g. Savani). However, current techniques are unable to effectively mitigate unwanted distillation at scale without major tradeoffs in performance. Thus, frontiers for future work include APIs that can detect and handle unwanted distillation events and improved techniques for anti-distillation sampling.

Despite their wide use, modern AI systems sometimes produce incorrect statements, accidentally or deliberately. In some cases, mechanistic interpretability techniques can determine what the system assesses to be true or false (Marks), in which case dishonesty can be defined as stating something ‘believed’ to be false, while incompetence can be defined as stating something false that is assessed to be true (Ren-B). In other cases, researchers lack insight into what, if anything, the model assesses to be true, rendering the honesty concept operationally undefined.

Dishonesty includes examples of AI systems providing users with information that is clearly false because it helps them achieve a broader goal (e.g. Scheuer). Methods for reducing the occurrence of false or confabulated generations (e.g. hallucinations) from systems are an ongoing research challenge. This can include both developing more truthful models through training on appropriate data (Evans-A) or designing systems to substantiate claims and cite references (Zhou). Frontiers for future work will include work to study and improve both factuality and honesty while also balancing these paradigms with risks of providing harmful information (Ren-A, Ren-B).

Model editing techniques offer an alternative to traditional fine-tuning methods by allowing engineers to make specific, precise changes to an AI system. By inducing targeted changes into the model, editing approaches can potentially offer efficiency and generalisation advantages over fine-tuning approaches (Wang). For example, model editing techniques might be useful for targetedly updating AI systems to correct an unwanted tendency such as hallucination or sycophancy. However, current tools are limited in their effectiveness and competitiveness. Frontiers for future work include improving both the scalability and efficacy of editing tools.

It is challenging to ensure that AI systems cannot cause harm when they have powerful capabilities. There is a broad space of AI capabilities, and risks generally increase with high autonomy, high generality, and high domain intelligence.

For example, AlphaFold (Jumper) has high intelligence in the narrow domain of protein folding, but lacks autonomy or generality (“A” or “G”). A robotic lawn mower has high autonomy but lacks generality or intelligence (“G” or “I”). Both are easy to control. A hypothetical future self-driving car that outperforms any human driver would have high autonomy and intelligence, but it poses a negligible loss-of-control risk due to low generality. Systems having all three traits “A”, “G” and “I” are the most difficult to align or control.

The three research directions below aim to improve trustworthiness by avoiding the “A”, “G”, or “I”, respectively.

1. Minimally-agentic systems: Techniques for training systems lacking agency/autonomy (no “A” in the above figure).
2. De-generalised (domain-scoped) systems: Unlearning, domain distillation, and other techniques for reducing/limiting domain generality to exclude risky knowledge and behaviours (no “G” in the above figure) (Li-B, Barez).
3. Intelligence-bounded (capability-scoped) systems: Techniques for reducing/limiting domain intelligence (no “I” in the above figure).

Several research programs aim to design AI systems in such a way that they are guaranteed to meet their specification, or they meet it with a probability that can be guaranteed. These include but are not limited to (Dalrymple et al):

• Verifiable program synthesis: Techniques for distilling subsets of machine-learned algorithms and knowledge into formally verified code (Michaud). This is also a form of capability reduction, as long as the algorithms and knowledge that an AI system is able to verifiably code up is a subset of all its knowledge. For example, a human physicist would typically be able to program a rocket steering algorithm they have discovered, but not the precise image-processing algorithm they use for recognising their mother.
• World models with formal guarantees: Research toward developing verifiable models of how AI systems affect their environment. These approaches range from probabilistic causal models to sound abstractions of physical laws, which remain challenging to construct but could enable precise reasoning about an AI's potential impacts.
• Compositional verification approaches: Developing methods to build systems from smaller verified components. This research direction aims to create verification chains from the hardware level upward.

These guarantees can also apply to only some modules or subsets of the system in order to reduce the degrees of freedom that need to be actively human-controlled.

The goal of robustness testing is to develop techniques for evaluating whether systems are trustworthy, even in novel contexts such as unprecedented “black swan” events or under attacks from malicious users. This includes developing improved red-teaming tools to identify inputs that cause systems to behave harmfully.

Adversarial robustness testing (security): Adversarial testing depends on techniques to evaluate system safety under deliberate attempts to make them behave harmfully. There are numerous approaches to adversarial testing involving attacks on the model through its inputs (“jailbreaks”/black-box attacks) (Jin), through API access (grey-box attacks) (Qi-A), and through its internal weights (white-box attacks) (Huang, Che). For example, researchers have developed many model “jailbreaking” techniques which can subvert the safeguards in modern AI systems, causing them to behave harmfully. Attacks can also be conducted on any data modality that the model can process. For example, multimodal models that can process text, images, video, and/or audio data can have a very large attack surface as a result. Key research goals include the continued development of more effective and scalable ways to attack systems and integrating those methods into evaluation frameworks. Over time, attack research must also adapt to new defences that emerge and vice versa.

Evaluating robustness in multiagent contexts: Systems that behave safely in simple, controlled settings can often fail in novel, more complex contexts. One very prominent version of this is emergent failures due to multi-agent interactions. These types of failures are expected to become increasingly prominent as highly autonomous AI agents continue to be adopted. For example, if one self-driving car learns to drive safely on streets with human drivers, it is still possible for it to be unsafe on streets with other self-driving cars because they may not behave exactly the same as humans. Multiagent failures are challenging to study because they often emerge unexpectedly and are hard to demonstrate in laboratory settings. Future work to study and identify emergent multiagent failure modes will involve a mix of theory, simulation, and field tests to understand emergent multiagent failure modes (Hammond). Further research should also study how agents deployed in the economy can communicate and cooperate with each other and with people and online services to avoid risks, e.g. through interoperability standards and agent authentication (Chan-A, Chan-B) as well as learning cooperative skills (Dafoe-A, Dafoe-B).

Techniques for quantitative, high-confidence assurances that an AI system meets its specification may be able to offer a strong potential foundation for developing safe systems. The special case of formal verification provides a 100% guarantee given specified assumptions.

Quantitative safety: Techniques that provide quantitative risk bounds could provide safety assurances akin to existing industry standards for systems such as jet engines and nuclear reactors. These solutions include formal verification approaches for proving that AI-written code, AI scaffolding, or AI containment measures meet precise specifications. Methods for quantitative safety and formal verification also include safe-by-design approaches. With current models and methods, it is not possible to use these techniques to make strong assurance about frontier system behaviours, but continued work may help to establish sound and practically useful techniques for making quantitative assurances of safety (Dalrymple).

Interpretability techniques aim to provide qualitative or quantitative evidence of system trustworthiness based on insights into why the AI system behaves the way it does.

Mechanistic interpretability: Techniques for understanding how models function and represent concepts internally uniquely allow for assessments of internal model cognition. These techniques could aid in the discovery of system properties or, if thorough enough, aid in constructing safety cases (Clymer, Buhl) for them (Sharkey). For example, mechanistic interpretability techniques might be able to help researchers characterise and intervene on model representations that correspond to harmful concepts such as deception or malice. Current research frontiers in mechanistic interpretability involve developing scalable techniques that beat black-box baselines for identifying and addressing flaws in systems. Mechanistic understanding of models could also help verify the success of other methods which are imperfect, such as unlearning of dangerous capabilities (see above) and analysing written ‘chains-of-thought’ (see below).

Explainability: Explainability techniques refer to methods that allow model behaviours to be attributed to specific features in their inputs. They can be useful for both diagnosing system errors and determining accountability for system failures (Gryz, Casper-B). However, current explainability tools are often unreliable (Bordt), highlighting the value of future work to improve on existing tools.

LLM chain-of-thought faithfulness and legibility: Large language model chain-of-thought reasoning does not always faithfully represent how a model arrived at its answer (Turpin). This poses challenges to safety because, without faithful reasoning, models could fool overseers by saying one thing and doing another. For example, language models have stated that they gave their answer based on a logical argument when they actually chose it based on hints that they should not have exploited (Anthropic-D, Turpin), such as seeing that the correct answer is always “B”. One potential challenge with chain-of-thought monitoring stems from how, under optimisation pressure on their reasoning, systems may learn to obfuscate their reasoning in ways that can be actively misleading (OpenAI-E, see also 2.1.1 above for an example).

Attributing model behaviours to training data: Methods for attributing model behaviours to specific examples from training data allow overseers to study how potentially harmful behaviours emerge in systems (Grosse). These tools could also help researchers identify what types of training interventions can mitigate them. For example, attributing control-subverting behaviours to specific examples from training data could help developers curate safer pretraining datasets. Research frontiers include improving the efficiency and scalability of these methods, causally studying how models develop personas and behaviours (Anthropic-F), and predicting what data is needed to learn a particular behaviour (Engstrom, Ilyas).

Studying goals in systems: Increasingly agentic AI systems are characterised by increasingly goal-oriented behaviour. As a result, studying the emergence and mechanisms behind these behaviours offers a way for researchers to study the system’s alignment with its specification (Ngo). However, it is challenging to study goals in AI systems because they cannot be inspected directly and their behaviour is sometimes but not always consistent with coherent principles (Khan, Mazeika). Directions for future work involve developing concrete definitions and measures of goals in AI systems (e.g. MacDermott) and interpreting how AI systems develop and represent goals internally (e.g. Marks).

Just as medical researchers use mice with induced diseases to safely study potential treatments before testing them on humans, AI safety researchers can create simplified AI systems to verify if safety methods are effective. These ‘model organisms’ are designed to allow controlled studies of specific safety issues that could emerge in more advanced AI systems (Hubinger-D). For example, researchers have created AI models with hidden "backdoors" that cause harmful behaviour only when given a specific trigger – simulating behaviour that malicious actors could insert or that models sometimes develop naturally (Greenblatt-A). This allows them to test whether safety measures can detect such behaviour (Marks). Companies can use this setup internally, but external parties can also create model organisms to audit the effectiveness of the safety methods used internally. Despite its importance, one survey study identified that this research area is still underrepresented (Delaney).

“Conventional” monitoring methods refer to techniques that can be straightforwardly integrated into many types of AI systems regardless of scope, domain, or intelligence. They often parallel techniques from other fields such as cybersecurity and content moderation. These techniques help researchers study systems and identify potentially harmful actions that systems might be taking. When incidents occur, these methods also help in the construction of incident reports.

Hardware-enabled mechanisms: Certain tools built into hardware can enable compute providers to know what is being run on their hardware. These techniques can help to monitor who is running what, where, and how much (RAND). Frontiers for future work on hardware-enabled mechanisms include both the engineering challenge of designing these tools to be efficient and the practical challenge of integrating them into compute infrastructure.

User monitoring: Monitoring for system misuse can help AI service providers identify potentially malicious users who may be seeking to misuse a system. It is a key part of “know-your-customer” approaches to risk management. User monitoring is not as simple as identifying potentially harmful instances of use (e.g. chats) due to (1) the risk of unintentionally impeding useful red-teaming (Longpre) and (2) the potential for adversarial users to implement sophisticated strategies to evade detection , such as using multiple accounts and obfuscation techniques. Frontiers for future work include iteration on methods to efficiently identify risky user behaviours with a low false positive rate.

System state monitoring: Techniques for monitoring a system’s activities can help to identify when it might be performing in a harmful or unexpected way. For example, a company providing a chatbot service may wish to filter the model’s responses using an unsafe-text classifier before sending them to a user. There are many different approaches that can be taken to state monitoring. Techniques can vary by the object of monitoring which can be system inputs, outputs, chains of thought, and/or internal cognition. They can also vary by the type of monitor which can include filters, event-loggers, and anomaly detectors. Frontiers for additional research include studying (un)faithfulness in LLM chains of thought and (e.g. Turpin) iteration on methods that achieve both a high degree of monitoring efficacy and efficiency, as well as methods for distributed contexts.

Designing modular, easily monitorable systems: Methods for decomposing complex systems into easy-to-monitor components ****have the potential to improve oversight in two ways. First, they mitigate risks of situational awareness and strategic evasion of human oversight (Berglund, Anthropic-C) by separating a goal-oriented system into multiple subsystems focusing on narrow tasks with no direct knowledge of each other. Second, they allow for the information passed between these systems to be more easily monitored. However, to date, limited empirical research has been conducted on the controllability of different modular systems and monitoring setups. Frontiers for future work include the design and testing of safe modular systems and methods to have systems decompose complex tasks into simpler, more easily-monitored subtasks (Wen-B).

Intervention techniques complement monitoring tools by offering various strategies to act on systems in ways that reduce risks from harmful behaviours.

Hardware-enabled mechanisms: Tools built into hardware could be used to enforce requirements about what can be run and by whom on specialised hardware (RAND). For example, hardware mechanisms could be used to block or halt certain jobs from being run on hardware if they fail an authentication process. However, the main barrier to their use continues to be the engineering challenge of implementing and integrating these mechanisms. If implemented successfully, hardware-enabled mechanisms could play a unique role in verifying compliance, even for international agreements and across borders (Brundage, IAISR).

Off-switches: An “off switch” refers to a mechanism that allows for the effective shutdown of a system. Shutdowns can be challenging for multiple reasons, including the distributed nature of systems and the need to pass critical tasks (e.g. driving) onto specialised risk-mitigating systems after a shutdown. Key challenges centre on the problems of implementing and integrating reliable mechanisms. Additional challenges for off-switches could be posed by systems that actively take actions that prevent humans from shutting them down. These will be discussed in 3.1.3.

Override protocols: Intervention procedures that replace harmful systems or system outputs with safe ones offer a final failsafe against harmful behaviour. For example, systems making high-stakes decisions may require a human in the loop to make key choices, and it may be either the system or a human that prompts for human intervention. The principal challenge and barrier to the effective use of overrides lies in the design of systems that effectively balance efficiency with safety. Toward this end, it will also be useful to define measures for measuring to what numerical degree a system is under the meaningful control of human operators.

Incident and emergency preparedness and response: Research is needed on protocols for rapid response and reporting for incidents without introducing new vulnerabilities (Wasil-A, Wasil-B). Technical and organisational questions remain about isolating compromised components while maintaining critical functions. A significant challenge involves verifying emergency response mechanisms against human error and potential exploitation by increasingly capable AI systems.

A particularly challenging frontier in operational control involves developing techniques for controlling AI systems that are not only highly capable but may actively and strategically attempt to undermine control mechanisms. Unlike conventional methods which offer system-agnostic approaches to monitoring and intervention, this section focuses on research toward techniques for controlling systems that are potentially very powerful and may actively undermine attempts to control them (Hubinger-A).

Scalable oversight: This research studies techniques that may allow less capable systems to oversee and control more capable ones. Common themes behind scalable oversight techniques include affording less-capable overseers more computational resources (Evans-B) or having multiple highly-capable systems debate (Irving, Michael), though these might not scale to superhuman systems. Research frontiers include improving the reliability of oversight even when systemic errors are present, developing more effective AI debate protocols, methods for monitoring systems for control-undermining behaviours, and methods for monitoring untrusted smarter systems with trusted weaker systems (Greenblatt-A). Another promising approach to oversight research is to combine multiple control strategies and attempt to quantify their joint success probability. For example, researchers can work to quantify the reliability of nested scalable oversight regimes where less capable systems oversee more capable ones (Engels).

Corrigibility and agent foundations research: Corrigibility focuses on techniques for ensuring that powerful agentic systems will allow an authorised user to change its goals or shut it down (Soares), as well as on the extent to which recursively self-improving systems will tend to retain their goals. Corrigibility can be challenging to implement because it is difficult to theoretically ensure that an agentic system has no incentive to interfere with such interventions (Omohundro). It is also theoretically difficult to control and predict how goal-oriented agentic systems may pursue self-modification, and how this affects goals (Everitt). This motivates theoretical and empirical work to study and control how highly agentic systems develop tendencies and behave in corrigibility and self-modification scenarios (Anthropic-F). Agent foundations research also explores how idealised AI agents would approach moral and game-theoretic reasoning (Hammond), including how AI systems’ behaviour can vary substantially depending on whether the system infers it is in a simulation versus in real-world environments (a phenomenon already observed in current AI systems (Anthropic).

Containment: As AI systems become increasingly agentic and able to access the internet, secure containment of risky systems is a growing challenge. Meanwhile, external threats to obtain unauthorised access to models also pose a security threat (Nevo). Frontiers for research on containment will involve developing techniques and protocols to keep high-security systems safe from internal and external threats.

Non-agentic AI systems as guardrails: Agentic AI systems, especially future ones that broadly outperform humans, pose major additional risks compared to other AI systems (IAISR, Bengio-B). However, non-agentic AI systems could be used to monitor their proposed actions and intervene by blocking them if needed. These include:

• A “Scientist AI” (Bengio-B) could be used to estimate the probability that a candidate action from an agentic AI violates any plausible interpretation of safety specifications, and reject potentially unsafe actions. Certain approaches could guarantee that unsafe actions will be identified, with quantitative error bounds (Bengio-B).
• “AI control” setups that can effectively monitor and intervene even if the underlying untrusted AI system is actively attempting to subvert these safeguards (Greenblatt-A, Korbak, Griffin).

Human-centric oversight: Since humans should ultimately remain in control of general-purpose agents, there will be some natural limits on what systems could be controllable in principle given the thinking speed, proactivity, expertise level, attention to detail, and reliability of human operators. Even with AIs assisting humans in understanding the given context in question per decision point or review point, Ashby’s Law (Ashby) within cybernetics suggests that the controlling system must have at least as much expressivity as the controlled system. Frontiers in research include how to formulate these measures.