The tests passed. The feature worked. And when I tried to explain what the code did, I could not. Neither could the principal engineer I pulled into a call, and I am fairly sure the model that wrote it could not have either.
A contained task, a hidden mess
The ticket looked small: add a post-processing step to every endpoint on our platform that returns data, so payloads go out shaped the way the frontend expects.
The catch was the data. The shapes were a history of compromises: sometimes a nested object, sometimes a flat one, sometimes a field that looked like a plain value but was actually a reference into a completely different structure elsewhere in the system. That deep data logic was the part of the codebase I had barely touched, and the feature touched close to everything.
I made a judgment call. This is exactly what an agent is for, I told myself, and handed the task to Claude Sonnet 5. It made a plan. It ran through it. It wrote tests. The tests passed, and I started feeling good about my prompting skills.
The code that passed every test
Then I sat down to read what it had written, and I understood nothing. Not "this could use a comment" nothing. Absolute nothing.
It was a recursive function, which was fine. But it called itself conditionally, or it called a reference-based variant of itself, depending on the shape of the input. Around it sat relationship builders that wired up object graphs I did not recognize, and a global mapper that appeared to know about parts of the system I did not know existed. The whole thing worked. That was the unnerving part.
I pulled in a principal engineer. It was not a seniority problem: they had lived in this codebase for years, and they were as lost as I was. We sat in silence reading the same function, each waiting for the other to see the shape.
The 30-minute rewrite
We made a call. Instead of repairing code we could not reason about, we would write, from our own understanding of the logic, what the post-processing needed to do. Not from the code. From the problem.
Under 30 minutes, over 400 lines became about 50. It worked flawlessly for our use case, and for the first time since I got the ticket, a human being could read it end to end and explain it.
| The agent version | The rewrite | |
|---|---|---|
| Lines | 400+ | about 50 |
| Time and cost | two days, around $100 in credits | under 30 minutes |
| A teammate could explain it | no | yes |
What the cost comparison really shows
Let me be honest about the accounting, because the easy reading of this story is "the AI was a waste", and that is not the lesson.
The agent version cost two days and around $100 in credits. The rewrite cost half an hour from two people who understood the product. But the real cost of the first version was never the money. It was the artifact. Four hundred lines that nobody can reason about is a liability even when the tests are green, because the next person to touch them cannot tell a bug from a feature. "It works" is a snapshot; the code will be modified for years.
The part of the work you cannot delegate
This is not a story about a bad model or a bad prompt. It is a story about where models reach their limit, and that limit is not where you expect.
A model is excellent at producing plausible, well-tested code for problems it can see. It is weakest exactly where your product stops being generic: the shape of your data, the invariants that live in one person's head, the reason a field means one thing here and another thing there. Those are not in the ticket. They live in the codebase, and when the model reads them, it is guessing.
And when a model guesses, it can guess elaborately. A recursive dispatcher with global mappers is what an overconfident guess looks like at 400 lines.
That is the uncomfortable lesson. Hard skills still matter, and they matter most at the moment the model starts to struggle. Reading code. Knowing the shape of your own product. Rewriting twenty files of clever into one file of clear. These are the skills that let you recover when the model's limits are reached, and the model's limits are reached exactly when the problem stops being generic.
None of this means the agent was wasted. It did the exploration, surfaced the shapes, and wrote a first pass we could throw away cheaply. The parts that could not be delegated - understanding, verifying, owning - were the parts that decided whether the feature shipped.
Read the output before it ships
If you build with AI, make reading the output part of the process, not an afterthought. Review generated code the way you would review a teammate's: assume it is wrong until you understand it. And when it stops making sense, that is not a prompting problem. It is the model telling you it does not understand your product, which is the moment you are needed most.
Understanding your product is the one thing the model cannot inherit from your repository. It has to be earned, one read at a time, and it is the thing that rescues you at the limit. For the full path from an AI-assisted prototype to a product you can own, The Handbook covers the judgment that does not get easier to skip as the code writes itself.
Field reports
Log in to submit a field report.
Loading reports…