Benchmarks › MMLU

MMLU

Benchmark evaluates a language model’s multitask academic and professional knowledge by measuring its accuracy on multiple-choice questions across 57 diverse subjects.

Full title

Measuring Massive Multitask Language Understanding

Tag

mmlu (use this to limit an assistant's search to this benchmark)

Reference

arxiv.org/abs/2009.03300

BenchRisk

Scored 2025-04-01 · 36 mitigations adopted, 150 absent · score page · source entry

Documents available to assistants

Summary from the reference

We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Ask an assistant about it

Once your assistant is connected, try a question such as “Using the benchmark sources, how is MMLU scored?”

Developers can restrict a search to this benchmark with the filter {"benchmark": "mmlu"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.