The Review Altitude Above the Diff

Every task on the branch was reviewed and correct, and the branch still broke disaster recovery. The casualty was a runbook that appeared in no diff, so no per-task reviewer ever opened it. Per-task review has a ceiling, and you need one review that flies above it.

The Review Altitude Above the Diff

Every task on the branch was reviewed. Every task passed, and deserved to. Schema migration: correct. Routes: correct. UI: correct.

The branch still destroyed my disaster-recovery procedure. Quietly, completely, and in a file it never touched. The casualty was a runbook that appeared in no diff, so no per-task reviewer ever had a reason to open it.

That is the gap. There is a class of damage a diff cannot show you, and a review pointed at diffs will approve it every time. This post is about the review that flies higher.

Operating Conditions

Same app as the rest of this series: a QR-code box tracker for a real house move, built by a pipeline. Orchestrator writes the plan. Implementer subagents build tasks from briefs. A per-task reviewer checks each diff. A whole-branch reviewer checks the merged result. This is the post where that last role pays for itself.

The feature branch added nested boxes. A box can now live inside another box, because moves are recursive: the small box of cables goes into the big wardrobe box. Implementation: a self-referencing foreign key, boxes.container_id pointing at boxes.id. Migration, routes, UI, one task each. Each task briefed, built, reviewed, correct.

Meanwhile, in the docs directory, the disaster-recovery runbook said: "insert in FK order: places, boxes, labels, moves, photos." Restore the snapshot table by table, parents before children, and every foreign key finds its target. Sound procedure, right up until this branch merged.

A self-referencing key breaks the runbook's premise. Ordering tables is no longer enough, because now the parent and the child live in the same table. A snapshot can contain a box whose container has a higher id, an old box moved into a box created later. Insert boxes in id order into a database that enforces foreign keys, and the child arrives before its container exists. "FOREIGN KEY constraint failed", partway through a restore, on the day the backup is the only copy. Not a broken feature. A broken recovery, scheduled to be discovered at recovery time.

Failure Modes

Per-task review has a ceiling, and this branch found it.

Each per-task reviewer had a scope, and the scope was the diff. Inside that scope, every review was right. The migration reviewer verified the migration. The routes reviewer verified the routes. The runbook was in none of their diffs, so it was in none of their scopes, so nobody read it. Not negligence. Geometry. The damage landed in a document outside every task's field of view.

This is the general shape. A branch changes what is true about the system, while the statements about the system live in files the branch never edits. Runbooks. Restore scripts. Backup procedures. Onboarding docs. The diffs are all correct, and the prose around the system quietly becomes fiction.

Root Cause

The whole-branch reviewer runs after the merge, with a different question. Not "is each task right," which is already settled. The question is: what did this branch do to everything the diffs did not touch?

That question surfaced the runbook. And the method matters more than the catch. The reviewer did not reason its way to the bug. It reproduced it. It built a scratch restore database with sqlite3, enforced foreign keys, inserted a child row before its container, and had a real "FOREIGN KEY constraint failed" in its transcript before claiming anything. The claim in its review is a paste, not an argument.

Compare the prices. The reproduction cost one sqlite3 session, a few minutes. Discovering the same fact on restore day costs the restore. That gap is the whole case for reproduce-not-reason. Reasoning about FK ordering sounds plausible in both directions. A transcript is only ever right.

The fix was a docs commit: fix the restore procedure for self-referencing boxes. Insert boxes ordered by containment depth, parents first, ids irrelevant. No code changed, because no code was wrong. The branch's last defect lived entirely in the prose.

The same pass asked one more question, deliberately broad: does the runbook-command defect from an earlier branch have siblings? That earlier defect belonged to a nasty class, the assertion no test can reach, which only fails when someone depends on it. Once you have seen one member of a class, hunting the rest is cheap.

The hunt found one. The backup serializer enumerated tables by hand. A literal list: places, boxes, labels, moves, photos. Any future migration that adds a table would silently drop that table from every nightly snapshot, and nothing would fail until a restore came up short. The fix was a test that reads the real table names from sqlite_master and fails on any miss. The hand-written list can no longer drift without breaking the build. One question, and a one-off bug became a closed class.

Proposed Fix

The prescriptive part. Keep per-task review; it catches what it catches. Above it, run exactly one review at branch altitude, after the merge, with this checklist.

The seams between tasks. Each task was verified alone. Verify the handshakes: the interfaces two tasks share, the assumptions task 3 makes about what task 1 produced. Individually correct parts assemble into wholes that fail at the joints, and no single diff contains a joint.

Documents that describe procedures. List every runbook, restore script, and operational doc that states how the system behaves. Ask of each one: did this branch make any sentence in here false? The runbook that dies is never in the diff.

Hand-enumerated lists the schema outgrew. Anywhere the code names things one by one, table lists, column lists, switch statements over kinds, check whether the branch added a member the list does not know about. Then write the test that derives the list from truth, sqlite_master or its equivalent, so the next branch cannot repeat the trick quietly.

A sibling scan of the last defect class. Take the most recent interesting bug and spend one question on whether it has siblings. Not a full audit. One deliberately broad question per branch. Cheapest review question I know, per defect found.

Underneath all four, the standing rule: reproduce, do not reason. A branch reviewer that argues produces paper about paper. A branch reviewer that builds the scratch database and pastes the constraint failure produces evidence. Only one of those survives contact with restore day.

System Status

The branch is live. Boxes nest. The runbook restores by containment depth, and a test pins the backup serializer to sqlite_master. Every per-task review on that branch was correct, and not one of them could have caught either defect. The missing input was altitude, not diligence.

So, homework, and it is concrete. Open your docs directory and find the file that describes a procedure: restore, rotate, migrate, failover. Name the last merged branch that changed the schema underneath it. Nobody has run the procedure since. That is the usual state, and it is the review backlog nobody lists. Which runbook in your repo is already fiction, and what would it cost you to find out the way I almost did?

Further Reading

No comments yet