NoveltyJudgeBench
Ai2 benchmark, data and code for testing whether LLM judges can tell novel research ideas from less novel ones.
Overview
Ai2 published Old Ideas, Novel Problems on October 27, 2026, a benchmark (Hugging Face dataset allenai/novelty-judge-bench), data and code for testing novelty judges plus an automated pipeline for building new evaluation sets. The Human-Only set has 154 high-novelty and 145 lower-novelty ideas from ICLR 2026 peer reviews, and the Human+Generated set pairs the 154 high-novelty ideas with 154 LLM-generated ones. Ai2 reports that changing the prompt moved GPT-5.4 accuracy on AI-generated pairs from 93.5% to 40.9%, and that a simple Claude Opus 4.6 baseline (0.854 macro-F1, $2.59) beat Idea Novelty Checker (0.642, $30.66) and the AI Scientist evaluator (0.772, $82.05).
This is a sourced directory entry. Features and capabilities are reported in the linked sources; this entry does not document independent Aymorphic testing.
Key features
- Two datasets: Human-Only (154 high-novelty, 145 lower-novelty ideas labeled from ICLR 2026 reviewer comments) and Human+Generated (154 pairs)
- Automated pipeline for building new novelty evaluation sets
- Prompt-sensitivity tests: one wording change shifted Claude Opus 4.6 by 21 points
- Compares agentic and simple judges, including an agentic GPT-5.6 Sol judge at 0.78 macro-F1 versus 0.89 for a simpler Opus 4.6 setup
- Released with a paper (arXiv 2610.02022) and code at github.com/allenai/novelty-eval
Use cases
- Checking whether an LLM-as-judge setup for research-idea novelty is stable before relying on it
- Comparing novelty evaluators such as Idea Novelty Checker or the AI Scientist evaluator on shared data
Pricing
Free; dataset on Hugging Face and code on GitHub. License not stated in the announcement.
Pricing reflects the source record and may change. Confirm current terms with the provider.
Dates and sources
- Released
- 27 Oct 2026
- Record updated
- 3 Nov 2026
- Record last checked
- 3 Nov 2026