StateBench
Ai2's benchmark of how language models track changing entity states across long documents, in four difficulty levels.
Overview
StateBench is an Ai2 benchmark, posted November 3, 2026, that tests whether language models can track values that change across long documents. It has four levels: r-trivial (set once), aperiodic (reassigned), soluble (x = x + 5 style updates) and general finite (x = y chained dependencies, linked to the S5 permutation problem). Ai2 reports that frontier models tested (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna) fall to 13% accuracy or lower on the longest general-finite problems. Gated DeltaNet and hybrid models trained from scratch at 275M parameters generalized to about 250x their training length on the three easier levels.
This is a sourced directory entry. Features and capabilities are reported in the linked sources; this entry does not document independent Aymorphic testing.
Key features
- Four levels from r-trivial to general finite
- Frontier models score 13% or lower on the longest general-finite problems
- Olmo 3 7B and Olmo Hybrid 7B evaluated alongside frontier models
- 275M models trained to 256 assignments tested up to 64,000 assignments
- Dataset on Hugging Face (allenai/state-bench) and code on GitHub (allenai/state-bench)
Use cases
- Evaluating long-context state tracking in new architectures
- Comparing transformers against recurrent and hybrid models
- Stress-testing agents that keep running state across long sessions
Pricing
Free; open dataset and code.
Pricing reflects the source record and may change. Confirm current terms with the provider.
Dates and sources
- Released
- 3 Nov 2026
- Record updated
- 6 Nov 2026
- Record last checked
- 6 Nov 2026