Part 9 of Building My Agentic Workflow in Public. In Part 8 I promised the numbers from round two of testing my pipeline. Here they are. The biggest one turned out not to be tokens, it was how many times the machine stopped to ask me a question.
When I first measured an autonomous epic in early August, the bill was ugly: 9.5 million tokens for five features, about 12,600 tokens for every line of shipped code, and around 27 eval rounds where only 16 percent of the findings were real bugs. But the number that actually ran my life was a different one. The pipeline stopped to ask me about 20 questions per epic, and each one cost me 5 to 8 minutes, because each one arrived written in the machine’s own jargon and I had to translate it before I could even decide.
Twenty stops at up to eight minutes each is not autonomy. It is a job. The ceiling on autonomous building is not model quality and it is not token budget. It is unanswered questions, because a halted run burns nothing but also ships nothing until I come back.
So the last three retrospectives all attacked the same enemy from different sides. This post is my own record of what changed, as much for me as for you.
Retro 2: measure, don’t assume
Retro 2 made two changes.
First, I put Part 8’s rule to work. If a spec claims something about the code, it has to show the command it ran and the number that came back. No more “all the plugin files follow this pattern” without proof. Run the command, paste the count, then we talk.
Second, I changed where the review happens. To be clear, the epic still ships every feature. What changed is the feedback loop. I used to hand the pipeline all five feature specs up front and let it review them against each other as one big pile of text, before any code existed. One of those review rounds burned 1.5 million tokens and produced zero code. All talk, no build. Now the loop runs per feature: build one, evaluate that one against the real code it produced, carry what we learned into the next one. Same five features at the end, but every review has actual code to look at instead of predictions about it.
Receipt: tokens per shipped source line collapsed from about 12,600 to about 2,500, and the validating epics shipped with zero escaped defects. Questions barely moved though: 20 became 19. Cheaper building, same interruptions.
Retro 3: make questions rare
The insight that named this post: treat questions as a currency the pipeline spends, and make it expensive. An agent may ask me only when two things are true at once, its confidence is genuinely low AND the decision would claim a permanent record. Everything else gets decided, written down, and routed to test time, where execution can prove it right or wrong for free.
One more rule came out of a painful week. I was checking a privacy field for leaks. Seven review rounds, all green, and I felt good about it. Then I looked closer: my test data had no leaks in it to begin with. Of course every round came back green. The test could not fail even if the code was broken. So the rule now has two parts. First the question: if this bug existed, could this test even catch it? And then the proof, because I do not take the test’s word for it. We break the code on purpose, the exact way the bug would break it, and run the suite. If it stays green while the code is broken, the test was decoration. Only when I have watched it turn red, and then turn green again after the fix, do I trust the green.
Did retro 3 work? Here is the verdict. The next two epics stopped to ask me only 3 to 4 times each, and every stop happened where I had planned for one, not in the middle of a build. For the first time, I could type the prompt, leave the house, and come back to progress instead of a stuck run waiting on me. That was the moment walk-away building stopped being a wish.
Retro 4: make questions cheap
The remaining questions were rare but still expensive, 5 to 8 minutes of mechanism-dense text each. We measured the same question rewritten in plain language: under a minute, answerable from a phone. So the format became law. Every decision question must lead with four plain lines:
DECIDE: the decision, as a human concern, in ordinary words
RECOMMEND: the pick and its why, one breath
IF YOU ACCEPT: what happens and what it costs
ONLY YOU KNOW: the one thing that would flip it
And I did not leave the format as a polite request, because agents drift back to old habits. I wired a hook into the pipeline that physically rejects any question missing the four lines. The agent has to rewrite it before I ever see it.
I also gave the agents a simple test for when a question deserves me at all. Ask me only if one of three things is true: the answer lives in my head and nowhere else, the action is hard to undo, or the decision sets a rule others will follow. Everything else, the machine decides on its own and writes down what it decided.
Let me show what “hard to undo” means in practice, because it carries most of the weight.
Some decisions are cheap to reverse. Renaming an internal field is a one-line fix. If it turns out wrong, we rename it back. The agent decides those alone and just writes down what it did.
Some decisions are expensive to reverse. Picking a data format that three other components will build on top of is nearly permanent, because changing it later means changing everything built on it. Those come to me first, before anything depends on them.
And a good agent can turn an expensive decision into a cheap one: tuck the debatable choice behind one small module, and reversing it later means swapping that module, not rebuilding the system.
So the question is never “is this decision important.” It is “what would it cost us to change our minds later.” Cheap to change, the agent handles it. Expensive to change, I hear about it first.
Did retro 4 work? On the next real epic, planning threw 38 questions at me and I answered each one in seconds, from my phone, without opening a single file. The build itself stopped only 5 times. And the bill came in around 0.9 million tokens per feature, the cheapest I had ever recorded. Cheaper, faster, and I barely felt the interruptions. That is what “make questions cheap” bought.
The honest ledger
Part 8 promised a surprise about escaped defects. Here it is, and it cuts both ways.
The good news first. Across all these epics, every real bug we caught was caught the same way: by running something. A live run, real data, a nasty input. Not a single bug was ever caught by reading the documents one more time. Zero.
Now the bad news. One epic finished with every test green, and I shipped it. The first live run found two bugs anyway. One was a deprecated API parameter. The other was a bug that an offline test could never see, no matter how good the test was, because it only shows up against the live system.
So I changed the definition of done. A feature is not done when the tests pass. It is done when it survives a live run. And I no longer trust a green suite at face value: first we break a test on purpose and watch it turn red. If it cannot turn red, it was never really testing anything.
If I compress all three retros into one lesson, it is this. I did not need smarter agents. I needed fewer questions, cheaper questions, and a habit of letting execution answer everything else, because running the code is the one reviewer that never lies.
And one thing matters enough to say before the next post: the questions that never reached me did not disappear. Every decision the machine took on its own gets written down, and at the end of the run I receive the full list, sorted by how hard each one is to reverse. I read the hard-to-undo rows carefully, skim the easy ones, veto what I disagree with, and some of those decisions become the next round of tests. My judgment did not leave the loop. It moved to the end of it, where it costs minutes instead of interruptions.
And then something happened
With the question economy in place, I tried something I would not have dared in August. I typed one short prompt, and walked away. About four and a half hours later I came back to something that genuinely surprised me, and it is now reshaping the entire next round of my process.
That story is the next post.