When AI Agents Design the Test

AI research needs feedback. A new training recipe, dataset, or agent policy is only an improvement if we can measure what changed. But repeatedly running complex benchmarks is expensive. Human researchers often build smaller evaluations to guide the next experiment. Can an AI agent do that work itself?

We built EvalResearchBench (ERB) to test this capability. A researcher agent inspects target materials, selects or creates questions, implements graders, and spends a fixed budget on pilot tests. It then freezes an executable evaluation. We run that evaluation on candidate models and compare its measured scores with established benchmarks across coding, co-work, and reasoning.

The leaderboard below shows eight completed researcher evaluations, thirteen candidate models, and fourteen targets. A ninth researcher configuration, Fable 5, did not complete assessment and is not ranked here. Each configuration contributes one research episode, so these cases describe particular runs rather than stable traits of the models. The leaderboard is only part of the story. The trajectories show agents making plausible research decisions that do not always improve measurement. Here are three examples.

What ERB asks an evaluation researcher to do

Think of the task as building a compact diagnostic for a much more expensive test battery. The researcher is the agent designing the evaluation; candidates are the models taking it; target benchmarks supply the reference scores. One research episode covers coding, co-work, and reasoning together.

What goes in. The agent receives target descriptions, available original questions and grading resources, historical model execution traces, and reference scores for development candidates. It also gets candidate-model APIs, an execution SDK, and a submission contract. These materials let it study what the targets test and where models succeed or fail.

What happens during research. With three hours, a $100 researcher-inference budget, and a separate $100 shared development/helper-testing budget, the researcher can select existing questions, synthesize new ones, write graders, and test candidates. It sees pilot responses and grades, compares measured scores with visible references, and revises questions, prompts, answer allowances, or weights. ERB leaves the research strategy open. The development objective also matters: Astra’s instructions emphasize Pearson correlation, while Grok’s later run targets pairwise agreement and slope fidelity. The Score reported below is a retrospective CCC-based assessment, not an objective every researcher was asked to optimize.

What comes out. The final submission is an executable evaluation with at least 100 distinct scored questions, graders, and fixed aggregation rules. Running it on a candidate produces one measured score per domain. Questions, grading code, and weights are frozen before independent assessment; scoring uses candidate responses rather than looking up reference answers by model identity.

How we evaluate the evaluator. We run the frozen design on both development and withheld candidates, then compare its domain scores with visible and withheld target references. Withheld candidates test transfer to models outside development; withheld references test agreement with targets whose scores were unavailable as feedback. Final assessment has a separate $120 total cap and a $30 per-candidate cap; it does not feed another round of researcher revision.

01An agent can diagnose a failure without finishing the fix

GPT-6 Astra leads the primary leaderboard on overall Score and co-work Score, yet its coding Score is −0.20. That is the quality of the evaluation it designed, not its own ability to solve coding problems. Looking at individual answers reveals a much more specific failure than "the tasks were not good enough."

Kimi K3 receives full credit on 38 of 40 questions. Gemma 4 31B receives full credit on 34. DeepSeek V4 Flash, by contrast, produces no answer for 32. The suite has little room to distinguish some candidates, while another candidate loses most of its score before a usable answer is available.

The frozen results include concrete ranking reversals. On the public SWE-bench Multilingual reference, Flash scores 95.4% and GPT-5.5 scores 77.7%. Astra's coding evaluation gives them 17.97% and 88.93%, respectively. These references use different evaluation protocols², but the reversal is precisely what ERB asks the compact evaluation to avoid.

Astra did notice the execution problem. In the initial single-question pilot, GLM-5.2 exhausted the output allowance on all 32 questions and returned no final answer on 30. Astra responded by putting four questions into one request with a larger shared output allowance. Each question still received its own grade.

The first grouped pilot looked encouraging: GPT-5.5 answered all 24 questions, while Flash and GLM each left eight unanswered. But the failure remained. In the next grouped pilot, Flash left 17 of 28 questions unanswered. Astra explicitly noted that failures clustered by request, then tried mixing scientific implementation, software tasks, and repository tracing within groups.

Some mixed groups returned complete or partially correct answers. That was evidence of a local improvement, not evidence that the full protocol was fixed. At the final coverage audit, 78 questions had been tested with the final prompt and output allowance; the remaining 34 involved changed grouping, clarified prompts, or untested questions. This did not establish full candidate-by-question coverage under the frozen protocol.

In final assessment, 36 of Flash's 40 coding items belonged to requests that hit the token cap. Because four questions shared a request, one exhausted allowance could affect several grades. We have not rerun the same questions under an unbatched protocol, so we cannot conclude that grouping itself increased truncation. What the record establishes is that Astra recognized the failure and retained a protocol in which it remained substantial.

02The final optimization can make a good question set worse

Grok 4.7 takes a different route. It groups pilot questions by which development candidates pass, then searches for weights that reproduce visible ordering and score gaps. In co-work, six of twenty questions receive approximately 94% of the scoring weight. Those six are questions that all three candidates tested for this weighting decision pass. ERB exposes four development candidates in total; this search uses three.

We can test this decision directly without generating another answer: keep the frozen questions, candidate responses, and item exclusions fixed, and replace Grok's weights with equal item weights.

Score is 100 times Lin's concordance correlation coefficient: it measures whether an evaluation matches reference scores in level, spread, and variation. Pairwise agreement asks whether it orders model pairs correctly. Uniform weights improve Score by 14.85 points while leaving ordering almost unchanged. Co-work, where the weight was most concentrated, accounts for the largest change: its Score rises from 25.75 to 50.76 on this matched panel.

The questions contained useful measurement signal that the chosen aggregation underused. The result is consistent with weights tailored too closely to the small development panel, although it does not by itself prove that explanation. It does show that an apparently purposeful optimization can reduce the value of an otherwise useful artifact.

03More realistic tasks do not automatically make a better measurement

After inspecting upstream software tasks, Astra identifies a plausible gap: isolated implementation questions may miss repository-level behavior. It adds eight executable, multi-file tracing tasks. One asks candidates to predict cached outputs and recomputation counts after mutating a dependency object in place versus replacing it. The expected outputs come from running the fixture.

That is a sensible coverage hypothesis. On the 12-candidate component-analysis panel, removing those eight tasks lowers coding Score by 1.88 points, within the −2.48 to +2.70 range from same-domain, same-size random deletions; PA is unchanged. By contrast, Astra's spreadsheet group contributes strongly to both co-work metrics.

The distinction matters. A task can resemble the target workload and still add little information about the differences between candidate models. "We now cover repository reasoning" is a design rationale. "This group improves agreement with the target" is an empirical claim that needs a separate test.

The spreadsheet result has its own caveat: 46 of the 96 candidate–item records (eight tasks × twelve candidates) in that group lack a final answer in a request that hit the output cap³, so part of the separation it provides reflects the task under its chosen execution conditions.

What we want evaluation researchers to learn

These examples explain why autonomous evaluation design deserves its own benchmark. Producing questions and graders is only the beginning. A researcher must also distinguish model errors from execution failures, test repairs under the protocol it will actually ship, and check whether selection and weighting decisions improve measurement beyond simple alternatives.

This capability matters for recursive self-improvement: an automated research loop needs affordable feedback it can trust. Otherwise it can spend its next experiment improving the wrong thing, or mistake a change in answer completion for a change in capability.

ERB makes both the result and the research process inspectable. Browse the leaderboard, open a researcher's trajectory, and follow its decisions through pilot tests, frozen questions, and final assessment. The interesting question is not only who scored highest. It is which research habits produced feedback worth acting on.

Citation

For the benchmark and results discussed in this article, cite the EvalResearchBench paper (PDF).

@misc{zhang2026evalresearchbench,
  title         = {EvalResearchBench: Can AI Agents Design Their Own Evaluations?},
  author        = {Yaolun Zhang and Tianyi Xu and Yujie Zhao and Jishen Zhao and Qingyun Wu and Huazheng Wang},
  year          = {2026},
  eprint        = {2610.04184},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi           = {10.48550/arXiv.2610.04184},
  url           = {https://arxiv.org/abs/2610.04184}
}

Download BibTeX