ThinkingBox
Microsoft's sandbox and 507-task benchmark that grades agents by final database state, now reusable as RL rewards.
Overview
ThinkingBox, from Microsoft with a Hugging Face post dated October 22, 2026, evaluates agents on stateful business workflows by checking the final database state instead of individual tool calls. ThinkingBox-Bench has 507 tasks, each with a real backend, MCP tools and a simulated LLM user. ThinkingBox-Training turns pass/fail results into binary RL rewards; with full-parameter GRPO on Qwen3.8-27B (187 tasks, 50 steps, 24 H100 GPUs), pass@1 rose from 51.70% to 60.78% and estimated cost per dependable task fell from $24.36 to $8.00, while pass@20 slipped from 89.35% to 87.97%. Code is MIT-licensed and benchmark data is CDLA-Permissive-2.0.
This is a sourced directory entry. Features and capabilities are reported in the linked sources; this entry does not document independent Aymorphic testing.
Key features
- 507 tasks with real backends, MCP tools and simulated users
- Grades terminal database state rather than tool calls
- RL training pipeline: Jeffreys-posterior difficulty banding, GRPO on Qwen3.8-27B
- pass@1 51.70% to 60.78%; cost per dependable task $24.36 to $8.00
- MIT code, CDLA-Permissive-2.0 data, OpenEnv environment
Use cases
- Benchmarking agents on stateful business workflows
- Post-training agents with verifiable, deterministic rewards
Pricing
Free and open source (MIT code, CDLA-Permissive-2.0 data).
Pricing reflects the source record and may change. Confirm current terms with the provider.
Dates and sources
- Released
- 22 Oct 2026
- Record updated
- 29 Oct 2026
- Record last checked
- 29 Oct 2026