Would you send the reminder your agent just checked?

Imagine opening a draft payment reminder at the end of a long day. It's polite, it's addressed to the right customer, and the amount matches your spreadsheet: Pinecrest Realty, $1,800. Your agent says it checked every draft against the file before calling the job done. Would you send it?

We built our free kit so that the agent checks its own work, and we expected that check to catch problems like this. In one of our test runs, it didn't. How does a careful self-check approve a bill that is wrong?

What the check was actually asking

The kit we give away runs every job as a loop: the agent writes a plan, carries it out one step at a time, and then reflects, which means checking the result by using it before it says "done". We call it plan, execute, reflect, and it is in our free PER kit.

The reflect step in that version told the agent to recompute numbers from the input file and to check each draft's names, amounts and dates against it. The Pinecrest draft passes that comparison: $1,800 in the draft, $1,800 in the file. The number wasn't invented. It was copied faithfully.

The file was the problem. Pinecrest's row said $2,100 billed and $500 paid, and $2,100 minus $500 is $1,600, not $1,800. (Our post about this same invoice shows what happens when you ask for these reminders in one quick line.)

The instructions never spelled out the other question: does the source add up on its own? In one of the two loop runs on the invoice job, the reminder went through. The other loop run caught it, and both one-prompt runs flagged the row as a data problem.

A self-check is only as good as the questions it asks.

The fix

We changed the reflect step. Before checking the drafts, it now runs a small script that tests each row's own arithmetic (amount minus paid should equal the balance), holds any row that doesn't add up, and traces every dollar figure in every draft back to its row. The plan and execute steps were told the same rule up front.

Then we ran the whole comparison again, twice as large. The loop held Pinecrest's reminder back in all 4 of its invoice runs this time.

What the loop costs

The same comparison showed what the loop costs. On simple jobs, it did not pass more often than the one-prompt version: across all four of the kit's jobs it passed 15 of 16 runs to the one prompt's 14 of 16, and a gap of one run is noise. On the three simple jobs, each run took about a minute longer with the loop and used about 11 cents more at API prices, a bit more than twice the cost.

So what is the loop for? In our view, three things the pass rate doesn't capture: the kit can stop for your approval of the plan before it does the work (our test approved plans up front, so it didn't measure that), the reflect step is a place to put questions like "does the source add up?", and the notes it leaves are meant to help the next run of the same job. Our test started every run fresh, so it didn't measure that last benefit either.

Try this today

You don't need our kit for the lesson. The next time you ask an agent to check its own work, add one sentence: "Before you trust any number you copy from a file, check that the file's own math adds up, and list any row that doesn't for me to look at." For code, the same idea means asking the agent to run the thing, not just reread it.

Step 4 of the lesson runs one job through the loop on the kit's sample data and gives you the loop rules to paste into your own project. One note: the kit release that step installs (v0.1.0) is from before this fix. The fix is on the kit's main branch and will be in the next release. Until then, add the sentence above, and look at any row the agent lists before you use a draft.

The most useful thing the loop produced in this whole test was a bug report about the loop.

How we tested it

• Setup: Claude Code 2.1.312 in headless mode with the Claude Sonnet 5.5 model, each run in a fresh copy of the kit with no personal settings or memory. The loop runs used the kit; the one-prompt runs got the same task text in a single prompt with the kit's loop files removed. A script checked each run against every testable line of the task's own "done means" list.
• The jobs: the kit's four sample tasks: invoice reminders, sorting an inbox, turning one article into posts, and researching a list of 25 business leads.
• First round (October 28, 2026, kit v0.1.1): 2 runs per job per side, 16 in all. Each side passed 7 of 8. The loop's one miss was the Pinecrest reminder (1 of its 2 invoice runs). The one-prompt side's miss was a lead list with 11 of 25 rows; it said so and stopped.
• Second round (October 29, 2026, the fixed kit): 4 runs per job per side, 32 in all. The loop passed 15 of 16 and the one prompt 14 of 16. The loop's miss was a lead list with 12 of 25 rows; the one prompt's were a list with 24 of 25 and a list missing its notes, with two rows lacking a website. The loop held the Pinecrest row in 4 of 4 invoice runs.
• Cost and time, three simple jobs: first round $1.33 against $0.54 for 6 runs each (2.5 times); second round $2.43 against $1.11 for 12 runs each (2.2 times), and 38.3 against 27.9 minutes in total (1.4 times, or about 52 seconds more per run). Cost is the API-equivalent figure Claude Code reports; on a subscription you pay in usage limits instead. On lead research the two sides swapped places between rounds, so neither was clearly cheaper there.
• Limits: small samples, one model, and sample tasks that are not code. The write-ups, scripts and grader are public on the kit's experiment branch (EXPERIMENT.md and RERUN_v012.md).

Next in the series: They asked Claude Code for a code review. Then they checked the commits.