HumanitysLastExam
Benchmark indicates whether a model can answer expert-level academic questions across disciplines with precision and understanding, serving as a key signal of the model’s reliability for users who depend on it for rigorous, high-stakes reasoning.
- Full title
- Humanity's Last Exam
- Tag
humanitys-last-exam(use this to limit an assistant's search to this benchmark)- Reference
- agi.safe.ai
- Related links
- lastexam.ai
- BenchRisk
- Scored 2025-04-01 · 41 mitigations adopted, 145 absent · score page · source entry
Documents available to assistants
| Type | Source | Format |
|---|---|---|
| Website | agi.safe.ai | HTML · 459 KB |
Summary from the reference
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at [this https URL.](https://lastexam.ai/)
Ask an assistant about it
Once your assistant is connected, try a question such as “Using the benchmark sources, how is HumanitysLastExam scored?”
Developers can restrict a search to this benchmark with the filter {"benchmark": "humanitys-last-exam"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.