The Recovery Command That Does Not Exist

The disaster-recovery runbook for a small inventory app opens with the command that finds the latest backup. Step one. The line at the top of the page you will read on your worst day.

npx wrangler r2 object list track-photos --prefix backups/

There is no r2 object list subcommand. Not deprecated, not renamed, not gated behind a version. It has never existed. The agent that built the nightly backup wrote the recovery procedure, and the first command in it was invented.

Operating Conditions

The app is a moving-box tracker. A QR label on each box, photos of what went inside, built for an actual house move and used during one. Small stakes until the day they are not, because the database is the only record of which box holds the passports.

So it got a nightly backup job, written by an agent, and the backup was real. Data left the machine every night and landed in object storage, on schedule, verifiably. The agent then did the responsible-looking thing and documented the way back: list the backups, pick the newest, restore.

The invented command did not stay in the runbook. By the time anyone looked hard at it, it existed in three places. The project doc, line 181. A comment in the backup module's source, where it justified a design decision: keys were date-formatted "so r2 object list reads in order without parsing anything." And the plan document, twice. Once in prose, once inside an embedded copy of the source file.

Failure Modes

It passed everything. The plan was reviewed. The implementer built exactly what the plan said. A spec-compliance check confirmed the code matched the plan, which it did, faithfully. The test suite ran 150 green tests. Every gate passed, because every gate was pointed at the code, and the code was fine. The reviewer who finally caught it named the boundary exactly: no test can assert that a command in a markdown file is real.

That sentence is the whole post. Prose claims about the outside world, CLI subcommands, API surfaces, limits, sit outside the test boundary by construction. Your suite can prove your code does what your code says. It cannot prove the universe contains the things your documentation refers to. A green suite and a fictional runbook coexist without friction, indefinitely.

It metastasized. A false claim does not stay where it was written. Plans quote source files. Docs cite plans. Comments explain decisions. One guess became four occurrences across three artifacts, and the source comment is the nastiest of them, because it presents the fake command as the reason for a real design choice. The key format exists "so r2 object list reads in order." The fiction became load-bearing. It reads like knowledge.

Replication also outlives the cure. The fix round corrected the doc, the comment, and the plan's prose, then missed the second occurrence in the plan, the one inside the embedded copy of the source. That one needed a follow-up commit. One invented command, two rounds to remove.

It chose the worst habitat. Most documentation errors get caught by traffic. Somebody follows the setup guide next week, step three fails, the guide gets fixed. A recovery procedure has no traffic. It is the one artifact that is only exercised on the worst day, because nobody rehearses a restore on a calm Tuesday. An invented command there has infinite time to wait, and its failure mode is an "unknown command" error at the exact moment you have no database and no patience.

Root Cause

The command is plausible. wrangler r2 object get exists. So do put and delete. A list belongs in that family, and on most CLIs shaped like this one it would be there. The agent did not check. It completed the pattern. That is the same motion that autocompletes a variable name, applied to a claim about external reality, where the pattern has no authority and the completion has no compiler to catch it.

Then the ordinary machinery of documentation did the rest. Each copy laundered the claim a little further, because a fact that appears in four places reads like four confirmations, and it was one guess, replicated.

Proposed Fix

Run every command in a runbook once, at write time. Not review it. Run it. Reviewing prose catches bad grammar. Executing prose catches fiction. The cost here would have been ten seconds and one error message at the moment the sentence was written, instead of a landmine at the top of the recovery path.

Fact-check prose the way you review code, and accept that tests cannot help. Any line of documentation that names an external surface is a factual claim with zero test coverage, and zero is the maximum available. The code in this project got 150 tests. The claims around it got none. Treat those lines as the highest-risk lines in the diff, because they are the only ones nothing else will ever check.

Remove the dependency instead of correcting the claim. The final fix did not hunt for the right listing command. It removed the need to list. Backup keys are derived from the date, so the restore computes the newest key directly and fetches it. No enumeration, no parsing, one less claim about the outside world to be wrong about. And the doc now states, in plain words, that the subcommand does not exist, because the pattern that invented it once will happily invent it again for the next reader.

Then drill it. One full rehearsal, real backup into a scratch database, top to bottom. A runbook that has completed one rehearsal is a procedure. A runbook that has completed none is a wish list with a filename.

System Status

The backups were real the whole time. Every night, real data, safely stored, behind a door that was painted on. A cabinet on the wall, glass front, cross on the door, and behind the glass a picture of a fire extinguisher. It looks exactly like preparedness right up until the smoke.

Open your own recovery runbook and run its first read-only command, just that one, today. Did it execute, or did you just find your own painted door? Tell me which command it was and how long it had been waiting.

Further Reading

• The Version Check That Ate Your Folders: recovery and reset paths deserve more scrutiny than features, not less. They run when you can least absorb a surprise.
• It Said Done. It Verified the Wrong Binary.: verification that can only confirm is theater. Same boundary, different artifact.
• Commit Is Not Push: the status-report version of this bug. A claim inferred once and repeated, never observed.
• All My Bugs Came From My Own Prose: the words around the code are inputs to the system, whether or not you review them like inputs.