GPQA
Benchmark indicates how well a system can answer graduate-level multiple-choice questions in biology, chemistry, and physics, with performance comparable to skilled non-experts even when provided unrestricted internet access.
- Full title
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Tag
gpqa(use this to limit an assistant's search to this benchmark)- Reference
- arxiv.org/abs/2311.12022
- BenchRisk
- Scored 2025-04-01 · 41 mitigations adopted, 145 absent · score page · source entry
Documents available to assistants
| Type | Source | Format |
|---|---|---|
| Paper | arxiv.org/abs/2311.12022 | PDF · 603 KB |
Summary from the reference
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.
Ask an assistant about it
Once your assistant is connected, try a question such as “Using the benchmark sources, how is GPQA scored?”
Developers can restrict a search to this benchmark with the filter {"benchmark": "gpqa"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.