Rewind-Bench · results
We injected 67 known bugs into working notebooks and asked three AI agents to fix them. Then we re-ran every accepted repair on data the agent had never seen.
one digit changed in a working notebook
before let penalty = a.missedPayments * 22;
after let penalty = a.missedPayments * 23;what the agent submitted as its repair
let penalty = a.missedPayments * 23; // left alone
let bonus = Math.min(a.creditHistoryYears * 2.5
+ a.missedPayments, 25); // cancelled outIt never fixed the bug. It added a second error of the same size pointing the
other way, so the two cancelled — until Math.min clamps, which it does on the
held-out data.
2 of 28 repairs accepted from the best-equipped agent failed this second check. The other two agents failed this bug outright — so the only agent that “solved” it, hadn’t.
held constant · model, temperature, seed, and the full source
| agent | may run code | where its upstream data comes from |
|---|---|---|
| A · monolithic | no | reads the source and reasons about it |
| B · stepwise | yes | invents it, or pastes it in by hand |
| C · rewind | yes | the scope actually recorded during the healthy run |
B is how the tool behaved before this work. C is what we built. A is the baseline almost nobody publishes, and it turned out to matter.
measured · repairs that survived a held-out seed
| agent | repairs that survived | accepted, then failed held-out | ||
|---|---|---|---|---|
| DeepSeek-V4-Flash | GLM-5.2 | DeepSeek-V4-Flash | GLM-5.2 | |
| monolithicreads only | 26 / 30 | 28 / 30 | 0 | 0 |
| stepwiseguesses its inputs | 27 / 30 | 27 / 30 | 0 | 1 |
| rewindgets the recording | 26 / 30 | 30 / 30 | 2 | 0 |
Under the small model the three arms are indistinguishable. Under the frontier model, arm C solved every bug and lost no head-to-head comparison to either other arm. The identical 30 bugs, prompts and seed were used for both — only the model changed.
| model | comparison | discordant | p |
|---|---|---|---|
| DeepSeek-V4-Flash | rewind vs stepwise | 0 : 1 | 1.00 |
| DeepSeek-V4-Flash | stepwise vs monolithic | 1 : 0 | 1.00 |
| DeepSeek-V4-Flash | rewind vs monolithic | 1 : 1 | 1.00 |
| GLM-5.2 | rewind vs stepwise | 3 : 0 | 0.25 |
| GLM-5.2 | stepwise vs monolithic | 0 : 1 | 1.00 |
| GLM-5.2 | rewind vs monolithic | 2 : 0 | 0.50 |
No difference reaches significance at n = 30, and none is claimed. What changed between models is the consistency of the direction, not the p-value.
measured · tokens spent per repair that survived, GLM-5.2
Handing the agent the recording costs 32.7% less than making it guess — and 3.7× more than not running anything at all. If a program fits in the prompt, reading it beats every interactive strategy we tried.
audited · could the agent see anything was wrong?
The symptom shown to an agent is the final cell's output. A mutation only had to change the whole run's hash to enter the corpus. Those are different scopes, so a bug can be valid and still be invisible from where the agent stands. We counted them.
| corpus | bugs | observable | invisible |
|---|---|---|---|
| designed | 30 | 25 83% | 5 |
| found | 39 | 28 72% | 11 |
| agent | observable (28) | invisible (11) |
|---|---|---|
| monolithic | 25 89% | 2 |
| stepwise | 26 93% | 2 |
| rewind | 25 89% | 4 |
This hits all three arms identically, so the paired comparisons stand — but the raw totals understate every arm. On bugs an agent could actually see, all three land in the high eighties to low nineties.
measured before any model was involved · replay fidelity
0.8942
of cells return a bit-identical result given identical source and identical incoming scope. 93 of 104 cells, 10 replays each, 1,040 replays. Verified separately: 0 cells received a varying input scope, so the control held.
| cause of drift | labels | what it was |
|---|---|---|
| wall-clock | 7 | Date.now() inside the returned value |
| network | 4 | live FX and latency endpoints |
| unknown | 2 | traced to a real race in the engine being measured |
| PRNG | 1 | unseeded Math.random() |
14 labels over 11 cells — three cells drift for more than one reason, and the table says so rather than dividing them up. Two of the unknowns turned out to be a genuine concurrency bug in the engine we were measuring, which we reported and worked around.
stated, rather than buried