EvalResearchBench
Can AI agents design their own evaluations?
We ask agents to design, test, and refine efficient evaluations within a fixed budget. Their frozen evaluations are assessed against complex benchmarks in coding, co-work, and reasoning.
Leaderboard
Paper results · 14 targetsScore = 100 × CCC. Higher is better. Target-equal average across 14 benchmarks.
| # | Researcher | ||||
|---|---|---|---|---|---|
| Loading results… | |||||
Scoring & methodology
Score is 100 × Lin's concordance correlation coefficient (CCC), measuring agreement with reference scores in level, spread, and variation. Perfect agreement is 100; opposing variation can score below zero.
Pairwise agreement (PA) measures model ordering: a correct order receives 1, a measured tie ½, and a reversal 0; reference ties are excluded. Displayed as a percentage.
Results are averaged equally across targets: 6 development and 8 sealed targets. Domain columns show the corresponding target averages; Overall averages all targets rather than the three domain columns.
Eight completed researcher evaluations cover 13 candidates. Random120 covers 12 candidates. The incomplete Fable 5 run is not ranked. Download source data.
Benchmark setup
Researchers choose the questions, graders, weights, and testing strategy. The final evaluation is frozen before independent assessment.
- Inputs
- Source tasks, trajectories, and development references
- Research
- Inspect → design → test → revise
- Output
- A runnable evaluation with 100+ questions
- Assessment
- 13 candidate models · 14 target benchmarks · 3 domains
Run & submit
The codebase provides configuration, budget controls, and independent evaluation. Start with the repository's setup instructions.
Submissions are reviewed before inclusion. Keep API keys local; do not include them in shared artifacts.
Citation
If you use EvalResearchBench, please cite our paper, arXiv:2610.04184.
@misc{zhang2026evalresearchbench,
title = {EvalResearchBench: Can AI Agents Design Their Own Evaluations?},
author = {Yaolun Zhang and Tianyi Xu and Yujie Zhao and Jishen Zhao and Qingyun Wu and Huazheng Wang},
year = {2026},
eprint = {2610.04184},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arXiv.2610.04184},
url = {https://arxiv.org/abs/2610.04184}
}