Evidence & evals
Agent code quality decays under iteration, and the cheapest fix for it has been tested and does not work. Both of those are measured. This page is the measurement, followed by one record of the same standard applied to us.
What the benchmark measures
Section titled “What the benchmark measures”SlopCodeBench (Orlanski et al., Mar 2026) chains an agent across 93 checkpoints of extending its own prior code. It is the closest published proxy for maintenance work: the agent inherits its own decisions and has to keep building on them.
| Element | Setup |
|---|---|
| Task shape | Extend the agent’s own prior code, one checkpoint at a time |
| Checkpoints | 93 |
| Models evaluated | 11 |
| Specification | Fixes external behavior only |
| Tests | Hidden from the agent |
| Context carried between checkpoints | None |
| Human comparison panel | 48 maintained open-source repositories |
What it found
Section titled “What it found”| Measure | Result |
|---|---|
| Problems solved end to end | None, across all 11 models |
| Best strict pass rate | 17.2%, falling to 0.5% by the final checkpoint |
| Trajectories with degrading code quality | 80 to 90% |
| Cost per checkpoint | 2.9x growth, with no gain in correctness |
| Verbosity against the human panel | 2.2x, widening every iteration |
| Human panel across the same span | Flat |
The cost line is the one that reframes the problem. Spending more per checkpoint buys more output and does not buy more correctness, so the decay is not a budget question.
The cheap fix, tested
Section titled “The cheap fix, tested”Telling the model to write better code is the obvious first answer, and it is the one the paper closes off.
| Quality-aware prompting | Effect |
|---|---|
| Initial verbosity | Down by about a third |
| Rate of decay | Unchanged |
| Pass-rate subtypes | Statistically unchanged |
| Cost | Up to 48% higher |
Better instructions improve the first checkpoint and do not survive iteration.
What the paper shows, and what we claim
Section titled “What the paper shows, and what we claim”The authors name one intervention they did not test: enforcing structural discipline across checkpoints, through training or through tooling. Flow-Next is the tooling half of that sentence. Gates the agent cannot skip. A different model reviewing every plan and every implementation. A receipt keyed to the exact commit behind every claim of done. Specs, decisions, and evidence as files in the repository, so the next checkpoint inherits a record instead of a scrollback.
The paper does not test Flow-Next, and nothing here claims it does. It establishes two things: the decay is real and measured, and the cheapest alternative to a verification system does not work. The verification spine describes what we built on top of that.
The plan-sync skip gate we removed
Section titled “The plan-sync skip gate we removed”/flow-next:work spawns a plan-sync agent after every completed task, to check whether the implementation invalidated a downstream plan. Each spawn costs 70 to 90 thousand tokens. We designed, built, and fully tested a deterministic gate that would skip the spawn when a task provably could not have affected anything downstream.
Cross-repo validation replayed 27 completed-task scenarios from three external repositories against ground truth from the real plan-sync agent. The gate produced a false skip: a change that altered a downstream task’s meaning while touching no file and no symbol that task referenced. Its skip rate was 1 in 15, against a design threshold of 50 percent.
The gate was removed from the shipped CLI. The decision record says not to try again.
Our own evals
Section titled “Our own evals”The same standard applies to changes we ship. Two results worth publishing because they went against the design.
The rolling scheduler, chosen against a faster arm. A pre-registered three-arm eval compared work-loop architectures on the same specs. Rolling admission with an isolated worktree per task saved 52.1% of work-phase wall clock at quality parity, with zero uncontained correctness incidents, and shipped. A shared-checkout arm was faster still at 69.2% and failed quality parity: part of its speed was bought with roughly 30% thinner test artifacts, because making the declared-paths list the commit boundary structurally disincentivized new test files and constrained verify windows made test iteration costly. The per-task worktree pool removes both pressures, so the speed is not bought with under-testing. After a field window on three hosts it graduated into /flow-next:work’s default scheduler. Details: work.
PR bodies, judged blind. The make-pr body format was evaluated by blind judging against the previous format: the old body scored 7/10 on reviewer effort and 5/10 on trust; the shipped format scored 9 and 9. The budget it holds to (a must-review set of at most about 30% of the diff) comes from that eval, not from taste. Details: make-pr.
Reading the evidence in your own repo
Section titled “Reading the evidence in your own repo”Everything the pipeline produces is inspectable in any repo that runs it: specs with numbered criteria under .flow/specs/, review receipts under .flow/review-receipts/, and evidence JSON on every completed task. How work gets proven explains the system; a week of running it produces your own numbers.