Startups

A new optimization framework claims 2.5x gains over Claude Code and Codex — here's what the benchmarks actually show

Arbor, from researchers at Renmin University of China and Microsoft Research, uses a branching hypothesis tree to accumulate knowledge across experiments. The results are promising, but the caveats matter.

Written by Lena Armitage · Bureau Tech · Jun 19, 2026

The problem Arbor is actually solving

When an AI agent is tasked with improving a codebase, the standard approach is to change several things at once — chunking strategy, retrieval method, system prompt — and then check whether the overall score went up. The problem is attribution: you can't tell which change helped, which hurt, and which did nothing. Worse, agents running long optimization loops tend to overfit to development metrics, producing impressive-looking scores that collapse when tested on held-out data.

This is the gap that Arbor, a framework from researchers at Renmin University of China and Microsoft Research, is designed to close.

How Arbor works

Arbor introduces a two-tier architecture. A long-lived **coordinator** agent acts like a principal investigator: it never edits the target codebase directly, but instead maintains a branching hypothesis tree and decides which ideas to test next. When it wants to explore a hypothesis, it spins up a short-lived **executor** agent, places it in an isolated git worktree, and hands it a single task.

The executor implements the idea, runs evaluations, and reports back. The coordinator then writes the evidence — including failures — into the tree and propagates the insight upward to parent nodes. A failed experiment becomes a recorded negative constraint, not a forgotten dead end.

This mechanism, which the researchers call Hypothesis Tree Refinement (HTR), is what separates Arbor from standard agentic coding loops. Most agents rely on conversation transcripts for memory; on tasks spanning hundreds of turns, those transcripts overflow context windows and the agent loses the thread. HTR gives the system a durable, structured record of what has been tried and why.

Before any improvement is accepted, it must pass a merge gate: the coordinator spins up a fresh worktree and tests the candidate against a held-out evaluator. Only verified gains are merged.

What the benchmarks show

The researchers tested Arbor against Claude Code and Codex on an autonomous optimization suite drawn from real-world AI engineering tasks, and against systems including AI-Scientist and AIDE on MLE-Bench Lite, a machine learning engineering benchmark.

Arbor achieved more than 2.5x the average relative performance gain of Claude Code and Codex under the same resource budget. On BrowseComp, a search-agent optimization task, Arbor raised held-out accuracy from a 45.33% baseline to 67.67%. Claude Code reached 53.33%; Codex reached 50%.

The overfitting results are worth noting specifically. On Terminal-Bench 2.0, Claude Code posted a development score of 75 but dropped to 71 on held-out data. Arbor's development score was lower — 72.22 — but its held-out score was 77.36. That's the merge gate working as intended.

Arbor also showed cross-task transfer: an optimized search harness built for BrowseComp improved performance on two unrelated search tasks, HLE and DeepSearchQA, without any additional tuning.

Where it fits — and where it doesn't

Arbor is designed to sit on top of existing Git workflows. Its output is a standard git branch that can go through normal code review. The main repository is untouched until a developer manually promotes the code.

But the cost profile is real. Maintaining a long-lived coordinator and running multiple concurrent isolated worktrees requires meaningful compute and token budget. Co-author Jiajie Jin identifies the sweet spot as tasks with a clear, trustworthy metric, a genuine search space, and tolerance for a long time horizon — pipeline optimization, data synthesis, model training recipe tuning.

Teams should avoid it for real-time latency tasks, obvious single-line fixes, or any situation where the evaluation metric is suspect. That last point is not a minor caveat. The entire framework's value proposition depends on the evaluator being honest.

Jin's stated next step — extending HTR from single scalar scores to multi-objective Pareto optimization across accuracy, latency, and cost — would address one of the more significant practical limitations for enterprise deployment. That work is not yet published.

Key takeaways

Arbor splits work between a long-lived 'coordinator' agent that manages strategy and short-lived 'executor' agents that run isolated experiments in separate git worktrees — preventing the entangled changes that plague single-agent optimization.
On the BrowseComp search-agent task, Arbor improved held-out accuracy from 45.33% to 67.67%, while Claude Code and Codex stalled at 53.33% and 50% respectively.
A strict 'merge gate' requires every candidate improvement to pass a held-out test evaluator before it's accepted — a design choice that directly addresses the reward-hacking problem common in autonomous coding agents.
Token cost is the dominant expense: maintaining a long-lived coordinator that continuously manages the hypothesis tree and dispatches executors is computationally intensive.
The framework's quality ceiling is strictly bounded by its evaluation metric. As co-author Jiajie Jin told VentureBeat: 'If the metric isn't trustworthy, Arbor will just optimize toward an untrustworthy result faster.'

FAQ

What is autonomous optimization (AO) and why is it hard?

Autonomous optimization is the process by which an AI agent iteratively improves a software artifact — a codebase, a data pipeline, a model training recipe — through experimental feedback, without step-by-step human supervision. It's hard because standard agent architectures treat each attempt in isolation, losing accumulated knowledge, and because agents tend to overfit to development metrics rather than producing improvements that hold up on unseen data.

How does Arbor's merge gate prevent reward hacking?

Before any candidate improvement is accepted, Arbor's coordinator spins up a separate isolated environment and tests the artifact against a held-out evaluator that was not used during development. The improvement is only merged if it demonstrably raises the held-out score. This breaks the feedback loop that allows agents to optimize toward a metric without producing real-world gains.

What backbone models did the researchers use in their tests?

The researchers tested Arbor with Claude Opus 4.6, GPT-5.5, and Gemini-3-Flash as backbone models for the coordinator and executor agents. The strongest MLE-Bench Lite result was achieved when Arbor was equipped with GPT-5.5.

Is Arbor suitable for all engineering optimization tasks?

No. According to co-author Jiajie Jin, Arbor works best when there is a clear and trustworthy evaluation metric, a genuine search space with multiple plausible directions, and tolerance for a long time horizon. It is not appropriate for real-time latency tasks, obvious single-line fixes, or any task where the underlying metric is unreliable.

How does Arbor integrate with existing development workflows?

Arbor is designed to sit on top of existing Git workflows. Each executor runs in an isolated git worktree, and only verified improvements are merged into a per-run trunk. The main repository remains untouched until a developer manually chooses to promote the code, meaning standard code review and CI processes apply.

Citations

New AI optimization framework beats Claude Code and Codex by 2.5x on the same compute budgetArbor delivered more than 2.5 times the verifiable performance gains of standard AI coding agents across real-world engineering tasks while operating under the same resource budget.
New AI optimization framework beats Claude Code and Codex by 2.5x on the same compute budgetOn BrowseComp, Arbor improved held-out accuracy from 45.33% to 67.67%, while Codex and Claude Code reached 50% and 53.33% respectively.
New AI optimization framework beats Claude Code and Codex by 2.5x on the same compute budgetOn Terminal-Bench 2.0, Claude Code achieved a development score of 75 but dropped to 71 on held-out data; Arbor scored 72.22 in development but 77.36 on held-out data.
New AI optimization framework beats Claude Code and Codex by 2.5x on the same compute budgetJiajie Jin, co-author of the paper, stated: 'If the metric isn't trustworthy, Arbor will just optimize toward an untrustworthy result faster.'