ResearchClawBench: Why AI Science Agents Still Miss the Paper

ResearchClawBench's July 7, 2026 BenchLM mirror puts Claude Code first at 21.5 RADS across 40 scientific research tasks, with the strongest mapped model row, Claude Opus 4.8 in ResearchHarness, at 21.1. That is the useful part of the new benchmark and the uncomfortable part of it: the public leaders are clustered around 21 on a scale where the authors describe 50 as matching the target paper.

We added ResearchClawBench today as a display-only agentic benchmark. It now has a source snapshot, a refresh script, exact verification records for mapped model rows, and a standalone leaderboard page. The source of record is the official ResearchClawBench dashboard, backed by the public leaderboard JSON, ModelScope dataset, GitHub repository, and paper.

We did not add it to weighted rankings.

That refusal is the point of this post. ResearchClawBench is one of the better public attempts to measure autonomous science, but it is also a benchmark where the unit under test is not cleanly "the model." It is the model, the harness, the tools, the runtime, the research prompt, and the workflow discipline wrapped around the whole thing. We show that evidence. We don't pretend it is the same kind of evidence as a GPQA or MMLU-Pro row.

The task is not question answering

Most science benchmarks ask a model to answer a hard question. GPQA asks graduate-level biology, chemistry, and physics questions. FrontierScience asks research-level science questions. HLE includes expert science items inside a larger frontier exam. Those are hard, but the shape is familiar: read prompt, answer prompt, get graded.

ResearchClawBench asks for a different kind of behavior. Each task is grounded in a real published paper, but the target paper is hidden from the agent. The benchmark gives related literature and raw data, then asks the agent to produce a research output that recovers the paper's scientific contribution. The public suite has 40 tasks across 10 domains: Astronomy, Chemistry, Earth, Energy, Information, Life, Material, Math, Neuroscience, and Physics.

The difference matters. A model can know a field and still fail to plan a research process. It can write plausible methods and still mismatch the experiment. It can extract a signal from data and still miss the central claim. ResearchClawBench tries to score the whole loop rather than the final answer alone.

The benchmark authors call the scoring metric RADS. The simple interpretation is useful enough to keep on the page: 50 means the output matches the target paper, and 70 or higher means the output surpasses the target paper. On that scale, a leader near 21 is not "almost there." It is a system that recovers fragments of the result while still missing enough of the scientific core to be nowhere near paper reproduction.

That makes the benchmark interesting.

The current public table

We mirror the official Pass@1 leaderboard and round the model-page value to one decimal. The full snapshot keeps two-decimal scores, task counts, costs, durations, and domain averages. We map only rows that the source labels as ResearchHarness (...) to model pages. Native agent products stay on the benchmark page.

Table 1 Download CSV

Row	How we treat it	Pass@1 RADS	Tasks scored
Claude Code	Agent row only	21.5	40
ResearchHarness (Claude-Opus-4.8)	Mapped to Claude Opus 4.8	21.1	39
ResearchHarness (Claude-Opus-4.7)	Mapped to Claude Opus 4.7	20.7	35
ResearchHarness (GLM-5.2)	Mapped to GLM-5.2	20.7	39
ResearchHarness (Claude-Opus-4.6)	Mapped to Claude Opus 4.6	19.9	37
ResearchHarness (MiniMax-M3)	Mapped to MiniMax M3	19.8	38
EvoScientist (0.1.1)	Agent row only	18.8	40
ResearchHarness (Qwen3.7-Max)	Mapped to Qwen3.7 Max	18.7	34
Codex CLI	Agent row only	18.4	40
ResearchHarness (GLM-5.1)	Mapped to GLM-5.1	18.2	40

The first thing to notice is the ceiling. The top five rows all land between 19.9 and 21.5. The spread is real, but small compared with the distance to 50. This is not a leaderboard where one agent has solved scientific research and the rest are catching up. It is a leaderboard where the best public systems are failing in different flavors.

The second thing to notice is the agent/model split. Claude Code leads the full table at 21.5 while Claude Opus 4.8 leads the mapped ResearchHarness set at 21.1. Those rows are close enough that the ordering is less important than the warning label. Claude Code is an agent product. ResearchHarness is a benchmark harness wrapping a model baseline. Codex CLI, OpenClaw, Nanobot, ResearchClaw, and EvoScientist are also systems. Treating all of those as equivalent base-model rows would be tidy and wrong.

The ones that do science know how much the lab notebook matters.

Why only ResearchHarness rows reach model pages

The source leaderboard includes 28 agent rows. Twenty of them are ResearchHarness model rows that map cleanly to model pages we already track: Claude Opus 4.8, GLM-5.2, GPT-5.5, Gemini 3.5 Flash, Qwen3.7 Max, and others. Eight are agent products or research systems: Claude Code, Codex CLI, ARIS Codex, OpenClaw, Nanobot, ResearchClaw, EvoScientist 0.0.4, and EvoScientist 0.1.1.

Model pages are built around model identity. They can show display-only rows from external systems, but they should not silently convert a tool product into a base-model benchmark. Codex CLI using GPT-5.4 is not the same row as GPT-5.4 in ResearchHarness. Claude Code using Claude Opus 4.6 is not the same row as Claude Opus 4.6 by itself. The agent may carry planning logic, file handling, execution behavior, retry habits, and hidden affordances that matter as much as the model.

So the mapping rule is deliberately narrow:

Table 2 Download CSV

Source row type	What we do with it
ResearchHarness (Model-Name)	Show on the model page as agentic.researchClawBench
Claude Code, Codex CLI, OpenClaw, Nanobot, ResearchClaw, EvoScientist	Keep on the standalone ResearchClawBench page only
Pass@5 rows	Preserve in the snapshot, do not mix into the Pass@1 model score

This creates a mild annoyance: the full benchmark page and model pages answer slightly different questions. Good. They should. The full page asks which public agent system did best on the official table. A model page asks what evidence attaches to this model under a specific harness. Collapsing those two questions into one number is how leaderboards get pretty and useless.

Pass@1 and Pass@5 are separate claims

ResearchClawBench also publishes a Pass@5 table for six ResearchHarness rows. In that table, Claude Opus 4.8 rises to 29.8, Gemini 3.5 Flash to 26.9, Kimi K2.6 to 24.9, GPT-5.5 to 23.2, MiMo-V2.5 to 22.5, and GLM-5.1 to 17.1. Those numbers are useful, and the snapshot preserves them.

They are not the same benchmark value as Pass@1.

Pass@5 answers: if the system gets five attempts, how good is the best or selected output under the source protocol? Pass@1 answers: what does one run produce? For scientific agents, the gap between those questions is large. Research is stochastic and path-dependent. A different initial plan can send the agent toward a better experiment, a better search path, or a less embarrassing dead end. Five attempts can reveal latent capability; it can also hide operational unreliability.

The current model-page value uses Pass@1 because the public table has broader coverage: 28 rows instead of 6. It also aligns with the main result discussed in the paper and source dashboard. Pass@5 remains visible where the source publishes it, but we do not average it with Pass@1, replace Pass@1 with it, or let it leak into the model ranking machinery.

That is a boring data-policy sentence. It prevents a bad leaderboard.

What the domain split says

The official task list gives four tasks per domain. Our snapshot keeps domain averages for each agent row, which is useful because the overall score can hide different failure modes. Claude Code's 21.5 average, for example, is not uniform. In our July 7 mirror, it averages 30.2 on Astronomy, 32.3 on Physics, 27.5 on Math, 25.5 on Material, and 24.9 on Information. Chemistry lands at 9.3 and Neuroscience at 5.5.

That pattern should not be overread. Four tasks per domain is thin. A single hard task can distort a domain average, and ResearchClawBench is a benchmark of end-to-end rediscovery rather than a clean domain-knowledge exam. Still, the split is a useful reminder that "science agent" is an overloaded label. A system that can make progress on one paper-reproduction task may collapse on another because the data modality, experiment protocol, or domain prior changed.

The paper's failure taxonomy lines up with that reading. The authors describe failures around experimental protocol mismatch, evidence mismatch, and missing scientific core. Those are not interchangeable mistakes. Protocol mismatch means the agent does something that looks like the right kind of experiment but not the experiment that would support the target claim. Evidence mismatch means it cannot line up observations with the needed argument. Missing the scientific core means the output may be polished, even plausible, while failing to recover the contribution that made the original paper matter.

That last failure is the one current LLM interfaces are best at hiding. A polished report can feel like progress. ResearchClawBench is valuable because it asks whether the report recovers the paper, not whether the prose sounds like a paper.

The benchmark is hard in the right way

A benchmark can be hard for boring reasons. It can require private knowledge. It can hide arbitrary labels. It can punish formatting. It can ask questions whose answer key is more fragile than the model. Hardness alone is not a virtue.

ResearchClawBench is hard in a more useful way. It compresses the parts of scientific work that agent demos tend to skip: reading adjacent work, choosing a plan, operating on data, writing down a result, and being judged against a real hidden target. The benchmark does not prove an agent can run a wet lab or manage a large collaboration. It does test whether the current research-agent loop can reconstruct a bounded scientific result when the ingredients are available.

The answer, for now, is mostly no.

That "mostly" matters. A score near 21 is not zero. The agents are finding pieces. Claude Code's best domain averages are not random guessing. Pass@5 lifts some rows substantially, which suggests retries and search diversity matter. The benchmark is not saying the tools are useless. It is saying the public state of the art is still below the threshold where a lab should trust autonomous paper reproduction without tight human supervision.

This is the kind of result a benchmark site should like: clear enough to guide decisions, harsh enough to resist the demo, and specific enough to show where the next audits should look.

Why it stays out of weighted rankings

Our weighted rankings are intentionally narrow about what counts. Weighted rows need to be comparable across models, sourced for the exact model variant, and close enough to model capability that a reader can interpret the score as evidence about the model. ResearchClawBench clears the sourcing bar. It does not clear the last bar.

A ResearchHarness row is closer to a model baseline than a Claude Code row, but it is still an agent-harness result. The score depends on tool wiring, prompt structure, run budget, and the harness's ability to keep the model moving through a long task. That is exactly why the benchmark is useful, and exactly why it should not be blended into the same weighted agentic category as Terminal-Bench 2 or BrowseComp without more policy work.

We already have a display-only tier for this kind of evidence. We use it for external indexes, agent-product leaderboards, arena-style rankings, and benchmarks with valuable but awkward comparability. Display-only does not mean decorative. It means visible, source-backed, and excluded from the score that orders models overall.

The distinction is not academic. If we weighted ResearchClawBench directly, Claude Opus 4.8 would gain a model-ranking signal from a harnessed scientific workflow while Claude Code's 21.5 would either be dropped or misassigned. Both choices would need a theory of how much of the result belongs to the model. We do not have that theory yet. Publishing the data without weighting it is the honest move.

What we shipped

Everything we shipped is reproducible on purpose. A new refresh:research-claw-bench script fetches the official leaderboard, Pass@5 table, and task list from the ResearchClawBench dashboard. It writes src/data/research_claw_bench_snapshot.json, syncs one-decimal Pass@1 values for the 20 mapped ResearchHarness rows, and leaves native agent products in the external snapshot.

The verification layer marks those mapped rows as benchmark_exact, with the official dashboard and leaderboard JSON in the source guide. The benchmark description records the RADS interpretation, task count, domain count, paper link, and display-only rationale. The source map documents the policy: Pass@1 is the model-page value; Pass@5 is preserved separately; agent products do not become base-model rows.

This is the part readers rarely see, but it is the part that decides whether a benchmark page is a measurement surface or a vibes spreadsheet. A new benchmark is not "added" when somebody types a number into JSON. It is added when the source, refresh path, mapping rule, verification status, exclusion policy, and future maintenance note all agree.

We ran that path for ResearchClawBench. The result is useful evidence, and it is bounded evidence.

How to read the score

Use ResearchClawBench when the question is: can this model-agent setup sustain a scientific research workflow long enough to rediscover a real result? Do not use it when the question is: which base model knows the most science? For that, pair it with GPQA, FrontierScience, HLE, and domain-specific evaluations. ResearchClawBench sits after those tests in the stack, not instead of them.

The main number to watch is not whether the leader moves from 21.5 to 23. It is when a public agent crosses the 50 line on broad coverage, under a protocol that keeps the target paper hidden and the scoring rubrics stable. A 50-point score would not mean the agent is a scientist. It would mean a bounded version of paper reproduction had become public evidence rather than a demo claim.

Until then, the benchmark's strongest message is negative and useful: current agents can assist research, search around it, draft around it, and sometimes recover meaningful pieces of it. They do not yet reliably reproduce the scientific core of hidden papers.

That is a number worth tracking.