Benchmarks › BigBenchExtraHard

BigBenchExtraHard

Benchmark indicates how well a system can perform 23 novel tasks that probe advanced reasoning capabilities, each designed to be significantly more challenging than its counterpart in the original BIG-Bench (BBH) suite.

Full title

BIG-Bench Extra Hard

Tag

big-bench-extra-hard (use this to limit an assistant's search to this benchmark)

Reference

arxiv.org/abs/2502.19187

Related links

github.com/google-deepmind/bbeh

BenchRisk

Scored 2025-04-01 · 43 mitigations adopted, 144 absent · score page · source entry

Documents available to assistants

Summary from the reference

Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general reasoning capabilities and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: this [https URL](https://github.com/google-deepmind/bbeh).

Ask an assistant about it

Once your assistant is connected, try a question such as “Using the benchmark sources, how is BigBenchExtraHard scored?”

Developers can restrict a search to this benchmark with the filter {"benchmark": "big-bench-extra-hard"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.