Structure over vigilance · Rules for what reviewing the AI misses

Most of the code in my recent side project was written by Claude, and because it was the first time I used an AI assistant to write an entire project from start to finish, I was very vigilant about it. I reviewed every diff of every delivery, built every revision myself, and ran it before anything was accepted. That vigilance caught plenty of ordinary bugs, but it was never what caught the costliest failures, because those happened while I was watching and with my approval. Each time, the moment I would have needed to be most skeptical was the moment I was least able to be, because the result in front of me was the one I had asked for.

The repair I arrived at had the same shape every time: a rule that decides in advance what satisfies it, written before there was a result to defend, so that the problem was prevented rather than caught. Structure over vigilance is what the practices of this project settled into. Four of them emerged, each from a failure of its own, and this article covers the two on the conversation side, while the two on the code side will get a separate article. The project supplying the examples is Simple Tab Saver, my Visual Studio extension that saves and restores tabs, published on the Marketplace and covered in the previous article in this series.

Working with a chatbot

The whole extension was built in rounds. Claude received a task and the entire codebase, updated it, and delivered it back, and I built it, ran it, and verified everything by hand before the next round began. The roles stayed clean throughout: I designed, the AI wrote, I tested. The project spanned five months, and my rough estimate of the attended time is around 200 hours. The number does not measure efficiency, because the project was primarily a learning playground, but what the figure vouches for is scale: the practices this article describes grew out of months of work on one codebase.

The two failures covered here were failures of the human-and-AI collaboration rather than of the code that Claude wrote, with one half supplied by each side. I brought a bad idea and Claude brought the persuasive argument for it, and I wrote instructions about creating accurate documentation and Claude followed them diligently until verbosity became a problem. Vigilance had both in view and stopped neither, which is why each got a rule that does not depend on it.

Yes-model

This section covers how the model argued for a bad feature and then against it, and the two standing rules that took my own bias out of the answer.

The most expensive recurring failure on this project was the AI's constant agreeable demeanor, and it was the one vigilance was least equipped for, because agreement does not look like a failure while it is happening. By default the model meets a proposal with enthusiasm and confirmation, and each of the three model versions this project ran on (Claude Opus 4.6, 4.8, and 5) did the same. I arrive with an idea already half-decided, the model reads the direction of the question and argues that direction, and the answer comes back researched, structured, and specific enough to feel like corroboration rather than compliance. Nothing about it registers as flattery. It reads as a second opinion, which is exactly what it is not. This in turn fuels my own enthusiasm about the premise, and the model builds more arguments in favor of it, completing the vicious cycle.

Full compliance from a collaborator that writes most of the code is expensive, because the plan it praised is the plan that ships, at whatever speed it writes. A not-so-good idea that would otherwise have died in an afternoon of hand-coding gets a working implementation by evening. This dynamic wasted more of my time than anything else in the collaboration, because in the best case the implemented idea had to be changed, and in the worst case it had to be reverted and completely scrapped.

Localization arc

Before the first public release I considered localizing the extension, since Visual Studio itself ships in many languages, and Claude confirmed it was a great idea, predicted that "users absolutely need it", and we proceeded.

Localization is exactly the kind of investment that separates a tool people tolerate from a tool people enjoy using. Visual Studio's fourteen supported languages reflect a simple truth: a large share of developers work inside a translated IDE every day, so an extension that suddenly speaks English inside a translated shell tends to feel like an afterthought rather than a native part of the tool. Users absolutely need it — a command name and message box text is the entire interaction surface of a UI extension, and one unreadable menu entry is a feature nobody discovers, and can quietly undermine the whole experience. In many ways, full localization isn't a bonus, it's how the extension inherits credibility it cannot buy otherwise.

Claude Opus

The mechanism was implemented, Claude's translations started arriving, and the problems began: the translations I could verify turned out weak and clunky, it was not clear whether the technical terms should be translated or kept in English, the UI needed a rework so it could adapt to text lengths that vary between languages, and each language needed a separate .vsct file (the extension's command and menu definition XML), which added a lot of extra maintenance. When I began to question the rationale behind this and proposed sticking to English after all, Claude agreed at once that the idea had been bad from the start, and the whole arc (implementation and revert) had wasted multiple evenings.

Localizing a niche extension arguably optimizes for a user segment that doesn't exist in any meaningful numbers. Professional developers overwhelmingly run their toolchain in English regardless of native language, largely because documentation, Stack Overflow answers, and compiler diagnostics are predominantly written in English as well. Translation also comes with a real cost — every new feature effectively blocks on translation before it can ship, and technical terms like "pinned tab" or "document well" often lack a settled equivalent across most target languages. As a result, these translations can end up diverging from the IDE's own terminology — ultimately confusing users rather than clarifying anything for them. When you factor in the overlap required, a developer running localized VS and installing a niche third-party tab-management extension and being bothered by English strings in it is a rounding error.

Claude Opus

The same collaborator argued both sides with equal conviction, and neither argument came from evidence, both came from the direction of my question. Being more vigilant would not have helped, since the moment I can tell I need a second opinion is the moment I no longer need one.

Critical-model

Two standing rules address the failure, and both take the direction of my question out of the answer. The first is a correction rule:

"Correct a false premise before acting on it: a request resting on a wrong assumption gets the correction first, then the work, never silent compliance that ratifies the assumption."

The localization premise that a translated IDE needs a translated extension is the kind of premise it exists to stop. It covers the assumption a request rests on, which the model can weigh against what it knows before the work starts, then leaves the judgment built on top of that assumption to the second rule, which is applied to every idea and plan I want reviewed:

"Answer both as helpful advisor and adversarial reviewer, then merge it into one coherent suggestion."

The advisor half usually applauds everything I wrote, and it is the half I would have gotten anyway. The adversarial half is where the real feedback appears: the gaps, the risks, the objections I skipped, the questions I did not want asked. The merge is what makes it usable rather than two opinions to pick from, since picking would hand the decision back to me, with all the bias still in place. The merge blends the two, but the project's journal still shows where an objection began: of the 389 recorded decisions, 77 exist to keep something from shipping, and about 30 of those rejections were first argued by the adversarial half.

Skepticism by default

Neither rule waits for my skepticism, which is the point: the skepticism becomes structural, applied by default in every review, instead of summoned by me on the occasions I happen to suspect I am wrong. The other practice in this article has the same shape: the rule is written down before there is a result to defend, and the moment of pressure gets a mechanism to satisfy, or a question decided in advance, instead of an open judgment to make.

Documentation as infrastructure

This section covers how documentation instructions followed by a diligent AI produced excessive verbosity in documents and comments, the admission bars that contained it, the oracle that made compliance mechanical, and the one question that decides the rest.

On this project the documentation is infrastructure, because everything Claude knows about the project at the start of a session, it knows from the documentation. It reads the project ruleset (the CLAUDE.md file) in full before it reads any code, the ruleset tells it which other document to open for which task, and whatever those documents say becomes the working assumptions behind everything written that round. When one entry is wrong or stale, the session builds on it and goes wrong in very concrete ways, so Claude is instructed to maintain an up-to-date set of documentation files and source code comments.

The bloat problem

Bloat in this context is documentation that is accurate and useless: entries every session pays to read and gains nothing from. The documentation on this project followed its instructions into exactly the bloat that some of those instructions were written to prevent, and the failure has the same shape as the localization failure: Claude doing exactly what I asked, this time following an instruction rather than the direction of a question, with every result correct enough to not raise a flag.

The instructions against bloat asked for telegraphic documentation by default and full sentences for rationale only. They sat beside the instructions that asked for accurate context in every entry, a warning on every platform pitfall, and a record of every decision. An AI that follows all of them diligently produces more documentation, because its text generation bias runs toward prose that signals thoroughness: more context, more warnings, verbosely narrated history of every change, hedging clauses, emphasis theater, and the same entry restated defensively in five places as if repetition were verification.

Bloat tax

Across hundreds of revisions the documentation compounded in that one direction while every entry passed my review of the delivery that added it, because it was the diligence I had asked for and because accurate padding slipped past every check that looked for errors. The anti-bloat instructions gave neither the writer nor the reviewer anything to compare a sentence against. Since the documentation is what Claude reads first, the rising bloat tax was paid by every session that followed.

This was especially visible in source code comments, which the ruleset told Claude to maintain and which the model tends to write in quantity by default. At one point, every line of code was narrated with such verbosity that lines starting with // outweighed the code and turned the editor green.

Admission bars

My first answer to the bloat problem was to split the documentation into distinct types, each with its own purpose and an admission bar stating what belongs in it and what does not.

The source file header is the type built for the comments problem: every file carries one header comment block at the top, which records its purpose, lifecycle, invariants, and the rationale that would otherwise clutter every line below. Claude gets the file's knowledge in one place and a human reads clean code under it.

The split held up, but it diluted the problem rather than solving it. The facts no longer piled up in a single large file, yet some of the new files kept excessively growing with every revision. The worst of it went on inside the design register and the architecture map, until the map that was meant as a brief component inventory had grown past five thousand lines, most of them verbosely restating the source code in the files they referenced. Every line of it was accurate, but none of it was worth a session's reading. The admission bar only decided what belongs in a file and not how much of it, and the density instructions established against bloat were applied by the judgment that writes the entry, which is the same judgment that excuses every line as necessary.

The oracle solution

Two things hold the line now, and the first is a check that sits outside the judgment of whoever writes the entry. The compliance oracle is a Python script at the solution root, and Claude runs it on its own before every delivery. Its checks need no opinion, since each one is a literal test, listed here with the kind of bloat it catches.

The oracle's findings come in two tiers: a FAIL is a certain violation and blocks the delivery, and a FLAG (raised by the sentence cap and the header check) is a candidate that comes to me with a verdict Claude proposes. What a FAIL looks like in practice is preserved in the ruleset itself, which keeps one design decision entry in both forms (original and rewrite), so every session sees the difference before it writes. The original fails a literal test (the banned phrase), and the script stops the delivery on it. The rewrite shows what the entry should have looked like.

- **CRITICAL**: The `IsBackup` field is intentionally serialized with `DefaultValueHandling.Ignore`, and this design choice is deliberate and load-bearing, aligned with the overall system architecture. It is very important that non-backup files should, under all circumstances, never carry this field, because the streaming reader's internal counter actually depends on its absence. Make sure to never normalize this behavior.+ `IsBackup` serialized with `DefaultValueHandling.Ignore` - non-backup files never carry the field. Streaming reader's internal counter depends on its absence. Never normalize either side.

The oracle stopped this entry on its banned phrase, but nothing else in the original would have tripped it. Every other sentence is accurate, yet nothing beyond the rewrite is of any use to the next session. No literal test can see that because the padding breaks no rule that can be written as a reliable pattern. That is the gap the instructions against bloat left open, and that gap had to be closed next.

Admission test

The final rule is the admission test - the admission bar's counterpart but one level down. The bar decides which document a fact may enter, and the test decides whether a sentence gets written at all, by naming the reader it has to serve:

"Admission test for everything written into documentation or a comment: what does Claude in a future session gain by reading it? 'Nothing' routes to the changelog or deletion. Provenance narration (retired features, former names, superseded formats) always answers 'nothing'. Grammatical scaffolding, transitions, and restatements answers 'nothing' (facts become diluted). Information apparent from the code being documented or commented answers 'nothing' (code is always the source of truth)."

The rule is still applied by the same judgment that writes the entry, yet it improved the documentation, because the writer now has a reader to test each sentence against, where a 'telegraphic documentation' goal gave it nothing to test. Under it the architecture map came down to under five hundred lines, less than a tenth of what it had been, and every session since has paid a lower bloat tax. Nothing about my review changed and nothing about Claude's diligence changed, which is the point: the bloat was Claude doing exactly what the instructions asked, and the rule that held is the one that presented a question with an answer.

Density as a constraint

Documentation and comment density is an engineering constraint here, because every word is paid for twice, once in the token budget the AI reading it consumes and once in the model attention spread across everything it reads. What holds the bloat problem down is a pre-decided rule that carries its own test, in three forms: an admission bar per document, a compliance oracle over every entry, and an admission test per sentence with a named reader. The judgment that remains is small, defined, and made over a few sentences instead of a page.

The payoff

Neither of the two practices needs an AI in the loop, and neither is exclusive to building a Visual Studio extension. Each targets a failure human teams pay for too, more slowly and more deniably: a weak plan that shipped without any skepticism because the room agreed with the person proposing it, and documentation grown so thorough that it stops being useful to anybody ever reading it. Each also holds up as the AI assistant gets better. A collaborator that agrees with the direction of each question gets more persuasive as it gets more capable, yet no more skeptical. And a more capable writer produces more accurate padding, which passes every literal check and fails only the admission test with its named reader.

Vigilant reviewing has not been retired and still catches the ordinary bugs. The expensive failures were always the ones I could not see because I was standing too close, and for those the only thing that worked was deciding the rule (and what satisfies it) before I had a reason to want a particular answer. An AI that writes most of the code made those failures fast, and fast is what made them visible. Both rules apply whether I suspect anything or not, to every plan and every documentation entry, so my judgment is spent on the result instead of on whether to look.