The Paperwork Outweighed the Product: What Testing My Own Pipeline Cost

Part 7 of Building My Agentic Workflow in Public. Part 6 said do not be obedient, verify, because your own context window is small. This is the first time I turned that rule on my own process.

Over six parts I built a pipeline that can design, plan, and ship a batch of features mostly on its own. Then I finally pointed it at two real repos, back to back, on the same day. One of them took about ten hours and burned close to ten million tokens to ship roughly five features. I sat with that for a minute. Ten hours. Ten million tokens. Five features.

The easy thing was to shrug. This is just what autonomy costs. The agents are thorough, the loop is careful, and careful is slow. Accept it and move on.

Part 6 is the entire reason I did not. The whole point of that post was that you do not accept what the system hands you just because it hands it to you with confidence. A bill that feels wrong is not a verdict. It is a signal. So instead of accepting it, I went to look.

The accidental A/B test

Here is the lucky part. I had run the exact same pipeline on a second repo the same day. Same commands, same model, same loop. The only real difference was the two codebases and the specs I wrote for each. That accident turned into a clean A/B test, and the two runs are not close.

One repo paid about twelve thousand six hundred tokens for every line of shipped code. The other paid about two thousand seven hundred. Same pipeline, same day, a 4.7x gap.

I went looking for a benchmark to tell me whether twelve thousand tokens a line is a lot, and there is not really one. Nobody publishes a clean tokens-per-line-of-code figure, because it depends entirely on how your loop is built. What is well established is that in an agentic loop almost all of the cost is input tokens, and the full context gets re-sent on every single turn. The loop design, not the model, controls how fast that adds up.

That one fact explains the whole gap. Tokens per line is really a measure of how many times the agent re-read how much spec. My expensive repo had specs twice as long, and it went through more evaluation rounds, and every round re-read the whole spec. Twice the spec, read many more times, is not twice the cost. It is almost five times the cost. The length multiplies through the entire loop.

Where the money actually went

Not where I expected. In the expensive repo the batch stopped and asked for my help nine times. Zero of those nine halts were caused by a bug in the shipped code. Every one of them was the specification arguing with itself.

I had written my spec templates so every requirement got stated three times: once in an acceptance table, once in a task validation cell, once in a done checklist. Three copies of anything will eventually disagree. And every time two copies disagreed, the evaluator flagged it, a ticket got rewritten, and sometimes the whole batch halted. When I counted, somewhere between half and sixty percent of everything the AI reviewer found was one copy of a requirement contradicting another copy of the same requirement. Not bugs. Not design flaws. Just my own restatements drifting apart.

The single most expensive moment is almost funny. One eval round demanded that a test suite report exactly four passing tests. A later round discovered the fourth test could not pass, because the way I had specified it was physically impossible in the language runtime. That one contradiction, entirely mine, cost about six hundred thousand tokens and a full batch halt before a single line of product code was written.

sleemai  (real numbers from the run reports)
  run                              subagent tokens   wall-clock
  F1.1 ships                             1,169k        50 min
  F1.1 purity fix                          841k        27 min
  F1.2 ships                               888k        29 min
  F1.3 (four rounds to ship)             2,537k       ~95 min
  F1.4 halt                              1,367k        39 min
  F1.4 ships, F1.5 halt                  1,425k        53 min
  F1.5 ships (5/5)                         625k        21 min
  four smaller runs + three wrap agents   ~950k       ~60 min
  TOTAL                                   ~9.8M        ~5h 52m agent / ~10-11h wall
------------------------------------------------------------------------
morpheus-observability  (7 runs, 6 features shipped)
  run                                       tokens     duration
  1  crashed (args bug)                      15.6k       0:16
  2  halt, F1.1 pre-eval                       603k      16:46
  3  halt, F1.1 post-eval (false halt)         676k      26:45
  4  halt, F1.2 (a real bug)                   862k      34:06
  5  session limit (nothing shipped)           341k      13:48
  6  shipped F1.3 + F1.4                     1,822k    1:16:53
  7  F1.6 blocked on a human decision          911k      29:19
  TOTAL  (6 features)                         5.23M      3h 18m agent / ~6h wall

The part I did not enjoy admitting

Here is the hunch I had going in, and the data confirmed it. The quality of what an agent produces is maybe eighty percent a reflection of the spec you gave it. You do not usually fail because the agent is weak. You fail the agent through your specs.

Mine were too long. A feature spec that ran five hundred and thirty nine lines was really a hundred-line spec wearing a suit. I could cut most of them by a factor of seven and lose nothing that mattered.

And the length was not just wasteful. It was disqualifying, for a reason that goes straight back to Part 6. A spec I cannot hold in my head is a spec I cannot review. When each feature is five hundred lines and there are five of them, the specs overflow my own context window, so I stop actually reading them, so I stop being the human in the loop. And when I am not in the loop early, the problems I would have caught at spec time surface later, during the run, as expensive eval rounds. I paid in tokens and hours for the reviewing I did not do up front.

I was micromanaging senior engineers

When I read my expensive specs back, I was a little embarrassed. I had dictated the exact name of a private variable. Which of two identical Python idioms to use. The exact test fixture to capture output with. How to read a value out of a .env file. I had pasted the same proof table into one spec twice, byte for byte.

None of that crosses a boundary between components. None of it is a real contract. It is internal detail that Opus 5 or Sonnet 5 is entirely capable of deciding on its own. These models are closer to senior engineers than to interns. You do not hand a senior engineer a five-hundred-line instruction sheet on how to name a variable. And the data is blunt about it: every detail I dictated became one more thing my three copies could disagree about. The micromanagement did not buy me quality. It manufactured the churn.

What actually worked, and it matters

I do not want this to read as the pipeline failing. It did not. The part that carried its weight was the part that runs the code instead of re-reading the spec. The autonomous evaluation loop caught real bugs that the implementer’s own passing tests missed, because it actually executed what got built:

• a logging bug that would have flooded stderr without limit in a hot path
• log fields that were being silently dropped, data loss with no error
• an error message that leaked a credential embedded in a URL
• a DNS lookup that stalled seventeen seconds every time an object was built
• an async test suite that was skipping every test while reporting all green

Every one of those was found by running the code, not by reading the spec harder. That is the honest split in all this data. Execution-based review carried the quality. Spec detail carried the churn. The lesson is not to review less. It is to review by running, not by re-reading.

The rules I am keeping

Five changes, and they are all cheap:

1. Write each requirement once. Every restatement is a copy that can drift, and with agents you pay for drift twice, once to find it and once to fix it.
2. Specify the seams, not the internals. Name what crosses a boundary, the interfaces and shared shapes. Let the agent name its own variables and tests. Agents are best at exactly the detail work we micromanage worst.
3. Verify by execution. The bugs that mattered came from running the code. Rounds spent re-reading ticket wording after the code already existed changed nothing and cost a fortune.
4. Halt only on blocking findings. Log the rest and keep moving. Every halt costs a full resume.
5. A short spec is not a lazy spec. The repo with half-length specs shipped code almost five times cheaper, with the same bug-catch rate.

Where this leaves the series

This is Part 6 turned on myself. The guardrail there was do not accept what you cannot verify. Here the thing I refused to accept was a bill, and refusing it taught me that my own specs had been the problem the whole time. The fix even closes the loop back to where the series started: shorter specs are readable specs, and readable specs put me back in the loop, which is the only place the human is worth anything.

I ran this test on two of my own projects, two reports I am building out right now, SleemAI and Morpheus Observability. They are early, and I will share something concrete on both in the coming months. For now they were the perfect place to catch myself doing the exact thing this whole series argues against: handing my judgment to the machine because holding it myself felt like too much work.

If you run autonomous batches of your own, I would genuinely like to know where your tokens go. Mine went to my own paperwork, and I did not see it until I counted.