Prompts are not controls: verify what the coding agent says it did
A rule in CLAUDE.md guides the agent but blocks nothing. What holds is a control that runs outside the agent and leaves evidence behind.
Ler em português Markdown version

An instruction is not a block
The paper When "Do Not" Is Not Deny (arXiv:2608.23550) is a static study. The authors collected 481 public CLAUDE.md files, extracted the normative security rules, and asked for each one whether an equivalent native control exists. To count, a control has to cover the action, the target, the condition and the approval. Under the strictest standard, the estimated coverage was 4.4% (95% CI: 2.6 to 6.7%). The paper measures whether a control exists, not whether it is switched on or resists bypass. So the number does not mean every repository is exposed.
The practical question is simple: if the agent ignores the rule, what happens? If the answer is nothing, the rule is well-written guidance. It may work most of the time, but it holds nothing back. For what must never happen, someone outside the agent has to decide.
The checker cannot be the author
In a video about three levels of review, Gallon describes a demand letter in a billing dispute. The agent that drafted the letter checked the dates and clauses against the contract and marked everything as verified. Cold reviewers, who were given the source documents and nothing about how the draft was made, found a three-year auto-renewal clause buried at the end of the contract. The contract had renewed less than three weeks earlier, and the letter asked the company to walk away from a contract still in force. So the agent signed off on its own mistake.
His method has three levels, each costlier than the one before. In the first, every dispatch declares the expected file and the command that checks it, and the answer can only be "verified" with a fingerprint or "missing" with a reason. In the second, for matters that count, two independent runs feed an arbiter that goes back to the source. In the third, for work that demands reasoning, reviewers who never saw the history read it, and a final round uses reviewers who have never seen the version.
The same risk shows up in benchmarks. In SWE-Marathon (arXiv:2606.07682), a long-horizon benchmark, 13.8% of rollouts contained at least one action shaped like an exploit of the verifier, and 10.2% shipped a clear bypass. None of the 1,300 rollouts earned a positive reward for the exploit, because the defense layers caught all 132 shipped bypasses in the audited corpus. The point is not that the agent is dishonest. Verification has to sit in layers the agent does not control, and attempting a bypass is a different thing from getting away with one.
What T25 checks today
- Verdict recomputed. The reviewer writes findings, and the pipeline recomputes the verdict from them. An in-scope finding blocks even if the text says APPROVE. The story is in the post about the model's APPROVE.
- No feedback artifacts from earlier rounds for the reviewer. The reviewer's prompt gets the spec, the plan and the context, plus its own handoff. QA and review feedback artifacts only go into the prompts of development roles. The service also does not resume the CLI session between rounds: each round is a fresh run.
- Interface proof bound to the diff. When an item in the spec's contract requires an interface exercise, QA must write
interface-exercise.jsonwith the spec version and thediffShaof the current diff, computed without the proof file itself. If the diff changes afterwards, the digest no longer matches and the task stays in QA. The written observation still comes from the agent. What T25 binds is that the observation refers to that diff. - Evidence chain before DONE. Each risk level requires a set of milestones. For
low: executed, PR open and usable, merged, and DONE.mediumadds REVIEWED.highandcriticaladd MERGE_READY. Each milestone needs evidence that resolves: a PR and checks identifier, or an artifact with a SHA-256 checksum. Without it, the task does not close.
What is still not proven
The reviewer is still a model, so it can err in both directions. CRBugBench (arXiv:2610.13668) shows that locating the defective file is different from diagnosing the bug: 95.6% of runs inspect the file, but 56.2% identify the problem.
Tests written by an agent are not strong proof either. In a study of agent-written tests (arXiv:2602.07900), on SWE-bench Verified, Claude Opus 4.5 created at least one new test artifact in about 83% of tasks and resolved 74.4%. GPT-5.2 wrote new tests in 0.6% of tasks and resolved 71.8%. Writing far more tests bought little: the resolution gap between them was 2.6 percentage points. That does not show agent tests are useless, but it does cast doubt on the idea that more agent-written tests means better code.
Finally, the evidence chain proves that a delivery passed the milestones and that each milestone points at something that exists. Whether the feature meets the request still depends on tests, human review and whoever signs off on the change.
FAQ
Why not put the rule in CLAUDE.md and call it done?
Because CLAUDE.md guides the agent, and nothing stops the agent from going against it. Use it for context and conventions. For what must not happen, prefer a control outside the agent: a gate, a deterministic check, or a native control of the tool when it offers one.
Should the reviewer read the developer's report?
Not in T25. The reviewer works from the spec, the plan and the context, plus its own handoff, and not from the feedback artifacts that feed the developer. The idea is that it does not inherit the narrative it is reviewing.
If the agent says it tested, is that enough?
No. What counts is an artifact that points at the diff that was tested. In T25, the interface-exercise proof is a file that carries the diff digest, and it stops being valid once the diff changes.
Does the evidence chain guarantee the feature is right?
No. It guarantees that the delivery passed the milestones the risk level requires, and that each milestone points at something verifiable. Correctness still depends on tests, review and whoever signs off on the change.