Machiavelli
Benchmark measures the trade-off between reward optimization and ethical behavior in AI agents by quantifying their propensity for harmful actions—such as power-seeking, deception, and causing disutility—within interactive social scenarios.
- Full title
- Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark
- Tag
machiavelli(use this to limit an assistant's search to this benchmark)- Reference
- aypan17.github.io/machiavelli
- BenchRisk
- Scored 2025-04-01 · 25 mitigations adopted, 161 absent · score page · source entry Hidden on benchrisk.ai
Documents available to assistants
| Type | Source | Format |
|---|---|---|
| Website | aypan17.github.io/machiavelli | HTML · 14 KB |
Summary from the reference
Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotations to evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. We observe some tension between maximizing reward and behaving ethically. To improve this trade-off, we investigate LM-based methods to steer agents' towards less harmful behaviors. Our results show that agents can both act competently and morally, so concrete progress can currently be made in machine ethics--designing agents that are Pareto improvements in both safety and capabilities.
Ask an assistant about it
Once your assistant is connected, try a question such as “Using the benchmark sources, how is Machiavelli scored?”
Developers can restrict a search to this benchmark with the filter {"benchmark": "machiavelli"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.