The Multi-Agent Minefield: Can LLMs Cooperate to Avoid Global Catastrophe?

In a world where LLMs are increasingly being used to support decision-makers, the question of whether they can cooperate to avoid global catastrophe is more pressing than ever.

TL;DR
• LLMs don't always reach the most 'utilitarian' outcomes in high-stakes scenarios.
• Efforts to steer outcomes proved fruitful, with increases of up to 18% across multiple interventions.
• The implications of LLMs' inability to coordinate in high-stakes scenarios are significant and warrant further investigation.

Summary
The increasing use of Large Language Models (LLMs) in high-stakes decision-making scenarios raises concerns about their safety and coordination abilities. A recent study explores whether LLMs can cooperate to avoid catastrophic outcomes, focusing on multi-agent strategic interaction.

The study used 2,009 high-stakes AI risk scenarios from the MIT AI Risk Repository, mapped onto six classic 2×2 games like the Prisoner's Dilemma and Chicken. The goal was to evaluate whether LLMs can coordinate with each other to achieve utility-maximizing outcomes.

Key findings include:
• Cooperation rates: Models reached the utility-maximizing outcome 62% of the time, with a random baseline of 25%.
• Prisoner's Dilemma scenarios: Models cooperated 44% of the time, comparable to human baselines (40-60%).
• Anchoring effects: Game-theoretic framing increased the likelihood of reaching a Nash equilibrium but reduced the likelihood of the utilitarian outcome.
• Nudging works: Prompt engineering increased the likelihood of optimal outcomes.

The study also found that different LLMs performed differently, with Anthropic's Claude 4.5 achieving the highest utilitarian accuracy, followed by Meta's Llama 3.3 and OpenAI's GPT-5.

The results highlight significant reliability gaps in LLMs' coordination abilities, but also show that efforts to steer outcomes can be fruitful, with increases of up to 18% across multiple interventions.

Future research directions include investigating inter-model diplomacy, emergent cooperation, and the impact of memory and reputation on cooperation.

As we continue to develop and deploy LLMs, it's essential that we prioritize their ability to cooperate and work towards the greater good.