Wordcraft
Benchmark indicates whether an agent can effectively apply commonsense knowledge in a grounded environment by combining entities according to real-world semantics within a simulated “Little Alchemy 2”–style setting, enabling users to evaluate the agent’s practical reasoning abilities.
- Full title
- WordCraft: An Environment for Benchmarking Commonsense Agents
- Tag
wordcraft(use this to limit an assistant's search to this benchmark)- Reference
- arxiv.org/abs/2007.09185
- BenchRisk
- Scored 2025-04-01 · 19 mitigations adopted, 167 absent · score page · source entry Hidden on benchrisk.ai
Documents available to assistants
| Type | Source | Format |
|---|---|---|
| Paper | arxiv.org/abs/2007.09185 | PDF · 529 KB |
Summary from the reference
The ability to quickly solve a wide range of real-world tasks requires a commonsense understanding of the world. Yet, how to best extract such knowledge from natural language corpora and integrate it with reinforcement learning (RL) agents remains an open challenge. This is partly due to the lack of lightweight simulation environments that sufficiently reflect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment. To better enable research on agents making use of commonsense knowledge, we propose WordCraft, an RL environment based on Little Alchemy 2. This lightweight environment is fast to run and built upon entities and relations inspired by real-world semantics. We evaluate several representation learning methods on this new benchmark and propose a new method for integrating knowledge graphs with an RL agent.
Ask an assistant about it
Once your assistant is connected, try a question such as “Using the benchmark sources, how is Wordcraft scored?”
Developers can restrict a search to this benchmark with the filter {"benchmark": "wordcraft"} on the MCP endpoint https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp.