The Reroll Trap: a debugging protocol for AI output that comes back wrong

You asked for something. What came back is wrong — not catastrophically, just wrong enough that you can't use it. So you do what everyone does: you regenerate. Maybe you add "please be more specific." Maybe you try a different model. Three rerolls later you have four mediocre versions, twelve minutes gone, and no idea which change helped.

That's the reroll trap. Regenerating is sampling from the same distribution that just failed you. It occasionally works, which is exactly what makes it addictive. Debugging is different: you find out why it failed, apply the fix that matches the cause, and the fix keeps working next month.

This protocol takes about a minute of thought before you touch the keyboard. Run it every time output disappoints you and your first-pass acceptance rate climbs within a week.

Step 1 — Stop. Name the gap in one sentence.

Before you type anything, finish this sentence out loud: "I wanted X, I got Y, and the difference is Z."

Most people can't. They feel dissatisfaction and reach for the button. If you can't state the gap, you cannot fix it — and neither can the model, because your next instruction will be as vague as your dissatisfaction. Half the time, saying the sentence out loud is the fix: you discover you never specified the audience, or the length, or that you wanted options rather than a recommendation.

Step 2 — Classify the failure. There are only four kinds.

Every bad output belongs to one of these classes. Diagnose by the tell, not the vibe.

Class 1 — Brief failure (you asked for the wrong thing)

Tell: the output is competent and confident but answers a slightly different question. It's a good essay about the wrong topic. If a smart colleague reading only your request would have produced the same wrong thing, it's a brief failure.

Class 2 — Context failure (it didn't know something it needed)

Tell: the output is generic, or it invents specifics. Placeholder names. Industry-average numbers where yours exist. Advice that would apply to any company on earth. Confident fabrication is almost always a context failure wearing a scarier costume.

Class 3 — Format failure (right thinking, wrong container)

Tell: the substance is fine but you'd have to rebuild it to use it. Prose when you needed a table. Nine paragraphs when you needed five bullets. A summary when you needed the email itself. This is the cheapest class to fix and the most commonly misdiagnosed as "the model isn't good enough."

Class 4 — Judgment failure (it made a call it wasn't qualified to make)

Tell: the output is well-structured and directionally wrong in a way only you can see, because the correct answer depends on taste, strategy, relationships, or private tradeoffs. It picked a vendor. It softened a message that needed to stay hard.

Step 3 — Apply the matching fix (not a bigger prompt)

Each class has exactly one right move. Adding more words to a prompt that suffers a context failure is how people end up with 800-word prompts that still produce mush.

Fix for Class 1 — Constrain the target

Don't re-explain. Add the three fields that were missing: audience, decision, and shape.

Fix for Class 2 — Feed the missing input, don't describe it

Paste the source. The actual numbers, the actual policy, the actual last-quarter report. If you find yourself typing a paraphrase of a document that exists, stop and paste the document. Then add one guardrail line:

If the same context keeps going missing across tasks, you have a standing problem, not a one-off. Build the reusable version — see the Context Pack.

Fix for Class 3 — Show the container

Describing a format is weak. Showing one is strong. Paste a past example and say "match this structure exactly, different content." One good example beats three paragraphs of formatting instructions, every time.

Fix for Class 4 — Take the decision back

Reissue the task as an options memo, not a recommendation: "Give me three options with the strongest argument for each and the condition under which each wins. Do not choose." You keep the call; you still get the prep work. Delegating judgment is the one failure you should fix by narrowing the job, not improving the prompt — the boundary line is drawn in the AI Delegation Playbook.

Step 4 — The two-fix rule

You get two diagnosed fixes. Not two rerolls — two fixes with a named cause. If the output still isn't usable after the second, one of three things is true, and you stop:

1. The task is under-defined even for you. Write the first paragraph yourself, then hand it back. You'll discover you were outsourcing the thinking, not the typing.
2. The task is actually two tasks. Split it. "Research the options" and "write the recommendation" fail together and succeed separately.
3. It's a human job. Do it yourself and note why, so you don't re-attempt it in six weeks.

The two-fix rule exists because debugging has diminishing returns too. Without a stopping rule you'll spend 40 minutes rescuing a 12-minute task, then call the whole approach a failure.

The worked example

The task: a monthly client update, historically 45 minutes of writing. First attempt returns a polished 700-word document that reads like a marketing agency wrote it about a company it has never met.

• Gap sentence: "I wanted a status update our client can skim in 90 seconds; I got a narrative essay with no numbers; the difference is specificity and length."
• Classification: generic + invented specifics = Class 2, with a side of Class 3.
• Fix 1 (context): paste the raw project log, last month's metrics, and the two open blockers. Add the "use only the facts above / list open questions" guardrail. Result: accurate, still too long.
• Fix 2 (format): paste last quarter's best update and say "match this structure exactly." Result: usable with two edits.
• The arithmetic: 45 minutes before. Now 4 minutes assembling inputs, 1 minute of prompting, 6 minutes reviewing — 11 minutes. And because both fixes were saved into the template, next month costs 8. Compare that to four rerolls, which would have cost 12 minutes and taught you nothing reusable.

The debug log (the part almost nobody does)

A fix you don't record is a fix you'll pay for again. Keep one file. Four columns, one line per diagnosed failure:

"Promoted" is the column that matters. A fix that stays in a chat window is a memory; a fix that moves into a saved template is infrastructure. Aim to promote every fix you apply twice.

Failure modes of the protocol itself

• Diagnosing by feeling. "It's just bad" is not a class. If you skip the gap sentence, you'll default to Class 4 and give up on tasks the model handles fine.
• Stacking fixes blindly. Applying all four fixes at once works occasionally and teaches you nothing — you won't know which one mattered, so you'll carry three useless instructions forever. One fix, one test.
• Prompt bloat. If your prompt has grown past a page and results haven't improved, you've been treating context failures with brief-failure medicine. Strip it back and paste real material instead.
• Blaming the model. Switching models mid-debug destroys your experiment. Diagnose first; if a properly-briefed, fully-contextualised task still fails, then model choice is a real variable.
• Debugging output you never defined as good. If you can't describe acceptable output in one sentence, you're not debugging, you're browsing. Build a standard first — a small eval set settles it permanently.

How you know it's working

Track three numbers for a month, on the back of an envelope:

1. First-pass acceptance rate. The share of tasks usable after zero fixes. Should rise as promoted templates accumulate. Under 30% means your briefs are the problem; over 80% means you're delegating tasks too easy to be worth measuring.
2. Rerolls per task. Should fall toward zero and be replaced by diagnosed fixes. This is the single cleanest signal that the habit has changed.
3. Repeat classes. If the same class shows up three times in your log, that's a structural gap — a missing context pack, a missing example library, a missing template. Fix the structure, not the instance.

The endpoint isn't perfect output. It's this: when something comes back wrong, your hand doesn't move to regenerate. It moves to the gap sentence — and you already know which of the four fixes you're about to apply.