Benchmarks › HumanEval

HumanEval

Benchmark indicates how well the system under test can support a human writing python code by specifying docstrings whose function bodies are completed by the system.

Full title

Evaluating Large Language Models Trained on Code

Tag

humaneval (use this to limit an assistant's search to this benchmark)

Reference

arxiv.org/abs/2107.03374

BenchRisk

Scored 2025-04-01 · 44 mitigations adopted, 142 absent · score page · source entry

Documents available to assistants

Summary from the reference

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

Ask an assistant about it

Once your assistant is connected, try a question such as “Using the benchmark sources, how is HumanEval scored?”

Developers can restrict a search to this benchmark with the filter {"benchmark": "humaneval"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.