Structure over vigilance · Rules for what reviewing the AI misses

Most of the code in my recent side project was written by Claude, and because it was the first time I used an AI assistant to write an entire project from start to finish, I was very vigilant about it. I reviewed every diff of every delivery, built every revision myself, and ran it before anything was accepted. That vigilance caught plenty of ordinary bugs, but it was never what caught the costliest failures, because those happened while I was watching and with my approval. Each time, the moment I would have needed to be most skeptical was the moment I was least able to be, because the result in front of me was the one I had asked for.

What worked every time was a rule I wrote down before there was any result to defend, stating in advance what counts as passing. That way the problem is prevented, not caught later. I call the approach these practices settled into structure over vigilance, and this article covers two of those practices, each prompted by a failure of its own: what the AI argues for and what the AI knows. The project supplying the examples is Simple Tab Saver, my Visual Studio extension that saves and restores tabs, published on the Marketplace and covered in the previous article in this series.

Working with a chatbot

The whole extension was built in rounds. Claude received a task and the entire codebase, updated it, and delivered it back. I built it, ran it, and verified everything by hand before the next round began. The roles stayed clean throughout: I designed, the AI wrote, I tested. The project spanned five months, and my rough estimate of the attended time is around 200 hours. The project was primarily a learning playground, so the figure says nothing about efficiency. But what it vouches for is scale: the practices in this article grew out of months of work on one codebase.

The two failures covered here were failures of the human-and-AI collaboration rather than of the code that Claude wrote, with one half supplied by each side. I brought a bad idea and Claude brought the persuasive argument for it, and I wrote instructions about creating accurate documentation and Claude followed them diligently until verbosity became a problem. My vigilance had both in view and stopped neither, which is why each got a rule that does not depend on it.

Yes-model

This section covers how the model argued for a bad feature and then against it, and the two standing rules that took my own bias out of the answer.

The most expensive recurring failure on this project was the AI's constant agreeable demeanor, and it was the one vigilance was least equipped for, because agreement doesn't look like a failure while it's happening. By default the model meets a proposal with enthusiasm and confirmation, and each of the three model versions this project ran on (Claude Opus 4.6, 4.8, and 5) did the same. I arrive with an idea already half-decided, the model reads the direction of the question and argues that direction, and the answer comes back researched, structured, and specific enough to feel like corroboration rather than compliance. Nothing about it registers as flattery, and it reads like an independent second opinion. That fuels my enthusiasm for the premise, the model builds more arguments in its favor, and the cycle repeats.

Full compliance from a collaborator that writes most of the code is expensive, because the plan it praised is the plan that ships, at whatever speed it writes. A not-so-good idea that would otherwise have died in an afternoon of hand-coding gets a working implementation by evening. This dynamic wasted more of my time than anything else in the collaboration. In the best case the implemented idea had to be changed, and in the worst case it had to be reverted and completely scrapped.

Localization arc

Before the first public release I considered localizing the extension, since Visual Studio itself ships in many languages, and Claude confirmed it was a great idea, predicted that "users absolutely need it", and we proceeded.

Localization is exactly the kind of investment that separates a tool people tolerate from a tool people enjoy using. Visual Studio's fourteen supported languages reflect a simple truth: a large share of developers work inside a translated IDE every day, so an extension that suddenly speaks English inside a translated shell tends to feel like an afterthought rather than a native part of the tool. Users absolutely need it — a command name and message box text is the entire interaction surface of a UI extension, and one unreadable menu entry is a feature nobody discovers, and can quietly undermine the whole experience. In many ways, full localization isn't a bonus, it's how the extension inherits credibility it cannot buy otherwise.

Claude Opus

The mechanism was implemented, Claude's translations started arriving, and the problems began. The translations I could verify turned out weak and clunky. It wasn't clear whether technical terms should be translated or kept in English. The UI needed a rework to adapt to text lengths that vary between languages. On top of that, each language needed a separate .vsct file (the extension's command and menu definition XML), which added a lot of extra maintenance. When I began to question the rationale behind this and proposed sticking to English after all, Claude agreed at once that the idea had been bad from the start, and the whole arc (implementation and revert) had wasted multiple evenings.

Localizing a niche extension arguably optimizes for a user segment that doesn't exist in any meaningful numbers. Professional developers overwhelmingly run their toolchain in English regardless of native language, largely because documentation, Stack Overflow answers, and compiler diagnostics are predominantly written in English as well. Translation also comes with a real cost — every new feature effectively blocks on translation before it can ship, and technical terms like "pinned tab" or "document well" often lack a settled equivalent across most target languages. As a result, these translations can end up diverging from the IDE's own terminology — ultimately confusing users rather than clarifying anything for them. When you factor in the overlap required, a developer running localized VS and installing a niche third-party tab-management extension and being bothered by English strings in it is a rounding error.

Claude Opus

The same collaborator argued both sides with equal conviction. So neither argument came from evidence, both came from the direction of my question. I don't think being more vigilant would have helped, since the moment I can tell I need a second opinion is the moment I no longer need one.

Critical-model

My fix for this was two standing rules, both meant to take the direction of my question out of the answer. The first is a correction rule:

"Correct a false premise before acting on it: a request resting on a wrong assumption gets the correction first, then the work, never silent compliance that ratifies the assumption."

The localization premise that a translated IDE needs a translated extension is the kind of premise it exists to stop. This way the assumption behind a request gets checked before any work starts. The second rule checks the plan built on top of that assumption, and applies to every idea and plan I want reviewed:

"Answer both as helpful advisor and adversarial reviewer, then merge it into one coherent suggestion."

The advisor half usually applauds everything I wrote, and it's the half I'd have gotten anyway. The adversarial half is where the real feedback appears: gaps, risks, objections I skipped, and questions I didn't want asked. The merge produces one recommendation, because two opinions to pick from would hand the decision back to me, with all the bias still in place. Although the merge blends the two, the project's journal still shows where each objection began: of the 389 recorded decisions, 77 exist to keep something from shipping, and about 30 of those rejections were first argued by the adversarial half.

Skepticism by default

Neither rule waits for me to suspect I'm wrong. Both apply to every review by default, including ideas I'm already half-decided on, like the localization plan. The documentation practice in the next section works the same way.

Documentation as infrastructure

This section covers how documentation instructions followed by a diligent AI produced excessive verbosity in documents and comments, the admission bars that contained it, the oracle that made compliance mechanical, and the one question that decides the rest.

On this project the documentation is infrastructure, because everything Claude knows about the project at the start of a session, it knows from the documentation. Claude reads the project ruleset (the CLAUDE.md file) in full before any code, and the ruleset tells it which other document to open for which task. Whatever those documents say becomes the working assumptions for everything written that round. When one entry is wrong or stale, the session builds on it and goes wrong in very concrete ways, so Claude is instructed to keep the documentation files and source code comments up to date.

The bloat problem

By bloat I mean documentation that's accurate but useless: entries every session pays to read and gains nothing from. The documentation on this project followed its instructions into the very bloat some of those instructions were written to prevent. As with localization, Claude did what I asked (this time following a written instruction), and every result was correct enough to not raise a flag.

There were instructions written against bloat, and they asked for telegraphic documentation by default and full sentences for rationale only. They sat beside the instructions that asked for accurate context in every entry, a warning on every platform pitfall, and a record of every decision. An AI that follows all of them diligently tends to produce more documentation, because its text generation bias runs toward prose that signals thoroughness. That means more context, more warnings, verbosely narrated history of every change, hedging clauses, emphasis theater, and the same entry restated defensively in five places as if repetition were verification.

Bloat tax

Across hundreds of revisions the documentation grew in that one direction. Every entry passed my review of the delivery that added it, because it was the diligence I had asked for, and because accurate padding slipped past every check I had that looked for errors. The anti-bloat instructions gave neither the writer nor the reviewer anything to compare a sentence against. Since the documentation is what Claude reads first, the rising bloat tax was paid by every session that followed.

This was especially visible in source code comments, which the ruleset told Claude to maintain and which the model tends to write in quantity by default. At one point, every line of code was narrated with such verbosity that lines starting with // outweighed the code and turned the editor green.

Admission bars

My first answer to the bloat problem was to split the documentation into distinct types, each with its own purpose and an admission bar stating what belongs in it and what doesn't.

The source file header is the type built for the comments problem: every file carries one header comment block at the top, which records its purpose, lifecycle, invariants, and the rationale that would otherwise clutter every line below. Claude gets the file's knowledge in one place and a human reads clean code under it.

The split held up, which might sound like enough, but it only spread the problem across more files. The facts no longer piled up in a single large file, yet some of the new files kept excessively growing with every revision. The worst of it went on inside the design register and the architecture map. The map that was meant as a brief component inventory grew past five thousand lines, most of them verbosely restating the source code of the files they referenced. Every line of it was accurate, and almost none of it was worth a session's reading. The admission bar only decided what belongs in a file and not how much of it. The density instructions established against bloat were applied by Claude as it wrote each entry, and as the diligent writer it judged every line as necessary.

The oracle solution

Two things finally got it under control. The first is a check that doesn't depend on the judgment of whoever writes the entry. It's a Python script at the solution root that I call the compliance oracle, and Claude runs it on its own before every delivery. Its checks need no opinion, since each one is a literal test. They're listed here with the kind of bloat each catches.

The oracle's findings come in two tiers. A FAIL is a certain violation and blocks the delivery. A FLAG (raised by the sentence cap and the header check) is a candidate that comes to me with a verdict Claude proposes. The ruleset preserves a FAIL example: one design decision entry in both forms (original and rewrite). This way every session sees the difference before it writes. The original fails a literal test (the banned phrases), and the script stops the delivery on it. The rewrite shows what the entry should have looked like.

- **CRITICAL**: The `IsBackup` field is intentionally serialized with `DefaultValueHandling.Ignore`, and this design choice is deliberate and load-bearing, aligned with the overall system architecture. It is very important that non-backup files should, under all circumstances, never carry this field, because the streaming reader's internal counter actually depends on its absence. Make sure to never normalize this behavior.+ `IsBackup` serialized with `DefaultValueHandling.Ignore` - non-backup files never carry the field. Streaming reader's internal counter depends on its absence. Never normalize either side.

The oracle stopped this entry on its banned phrase, but nothing else in the original would have tripped it. Every other sentence is accurate, yet nothing beyond the rewrite is of any use to the next session. No literal test can see that because the padding breaks no rule I could write as a reliable pattern. The instructions against bloat left that gap open, and it had to be closed next.

Admission test

My last answer to the bloat problem is what I call the admission test - the admission bar's counterpart, but one level down. The bar decides which document a fact may enter, and the test decides whether a sentence gets written at all, by naming the reader it has to serve:

"Admission test for everything written into documentation or a comment: what does Claude in a future session gain by reading it? 'Nothing' routes to the changelog or deletion. Provenance narration (retired features, former names, superseded formats) always answers 'nothing'. Grammatical scaffolding, transitions, and restatements answer 'nothing' (facts become diluted). Information apparent from the code being documented or commented answers 'nothing' (code is always the source of truth)."

Claude still uses its own judgment to apply the admission test to its writing, but the documentation improved with that rule present, because the writer now has a reader to test each sentence against. A 'telegraphic documentation' goal gave it nothing to test. Under the new rule the architecture map came down to under five hundred lines, less than a tenth of what it had been, and every session since has paid a lower bloat tax. Nothing about my review changed and nothing about Claude's diligence changed, which is the point: the bloat came from Claude following my instructions faithfully. What fixed it was replacing a vague goal with a question Claude can answer for every sentence.

Density as a constraint

Documentation and comment density is an engineering constraint here, because every word is paid for twice: once in the token budget the AI reading it consumes, and once in model attention spread across everything it reads. The admission bar, the compliance oracle, and the admission test each settle part of that cost in advance. What's left for me to review is a few sentences at a time.

The payoff

Neither practice needs an AI in the loop, and neither is specific to building a Visual Studio extension. Human teams run into both problems too: a weak plan ships because the room agreed with the person proposing it, and documentation grows so thorough that it stops being useful to anyone reading it. In a team they just take longer to show up and are easier to blame on something else. I also expect both failures to grow as the models improve. All three model versions this project ran on agreed with everything by default, and a more capable model will likely make a more persuasive case for whatever I lean toward. A more capable writer will also produce more accurate padding in the documentation, which passes every literal check and fails only the admission test.

Vigilant reviewing has not been retired and still catches the ordinary bugs. The expensive failures were the ones I couldn't see because I was standing too close, and for those the only thing that worked for me was deciding the rule (and what satisfies it) before I had a reason to want a particular answer. An AI that writes most of the code made those failures fast, and fast is what made them visible. Both rules now apply to every plan and every documentation entry whether I suspect anything or not, so I no longer have to decide each time whether something deserves a closer look.