PT EN
Install

Lights-on: why the software factory keeps a human at the merge

Teams that tried to take people out of the code ended up reading it again. Here is what T25 keeps at merge, at the risk gate and at the diff ceiling.

Ler em português Markdown version

An empty teal hangar with a lit gate at the far end and a single orange lamp on a console in the foreground.

Three sources that point at the same limit

Dex Horthy, from HumanLayer, gave the talk Why Software Factories Fail. His core argument is that a well-built harness is not enough. His team tried lights-off, and by his account agents started struggling after three to six months, on problems nobody was reading the code for anymore. He blames the reward signal. The agent is rewarded when the problem is fixed and nothing breaks. That signal does not penalize poor design or the erosion of maintainability, and the cost only shows up months later. His answer is to align before coding: architecture, then program design (types, signatures, call stacks), then vertical slices. And keep reading the code.

pasquadev answered the talk in a video that describes how his team works: a PRD reviewed with product, a peer review with other staff engineers before any code, and pull requests of around 300 lines. He also uses hexagonal architecture as a yardstick. The deeper a diff reaches into the domain, the more it needs a human. Near the edges, less. He says he is still thinking this through and has not built it.

The CRBugBench paper (arXiv:2610.13668) measures automated reviewers on pull requests that introduced known bugs. Its metric, recall@k, is the fraction of tasks where the known bug appears among the first k findings. With Claude Opus 5 on Copilot CLI, the paper reports 71.9% at top-5. That figure averages two sets: 79.6% of bugs found before merge and 64.2% found after. The same analysis shows the detail that matters. 95.6% of runs inspect the defective file, but only 80.2% produce a finding for it, and 56.2% actually hit the bug. Finding the file is not the same as diagnosing the bug. The paper also says reviewers lose effectiveness on complex changes and on bugs that need reasoning beyond the diff. Its recommendation is to keep a human responsible for consequential changes and for maintainability.

Corey Gallon, in another video, says roughly four of every ten sessions he reviewed contained an agent error that counted. His method is agent-based review in three levels that rise with the stakes. He admits he cannot check everything himself, so his design does not put a human on every step. That is where it differs from T25: here, the merge is not delegated.

What T25 does today

Control point How it works in T25 Where it lives
Spec Human approval. No policy promotion removes this step. src/decisions/action-promotion.ts
Plan Risk-based: low does not require plan approval by default, medium does, high requires plan and PR, critical always requires a human. Each project can turn the gate on or off. src/core/config.ts, requiresPlanApproval in src/core/policy.ts
Diff A ceiling of 40 files, 2000 lines and 0.5 duplicate-line ratio, all configurable. Over the limit, the task fails and gets a handoff with the numbers. limits in src/core/config.ts, checkDiffLimits in src/core/policy.ts
Sensitive paths review.sensitive_paths per project. A PR that touches those paths proceeds only with a human acknowledgement, and the acknowledgement is bound to the diff digest: if the PR changes, it asks again. merge in src/t25/service.ts
Merge Human. The factory opens the PR and checks the required status checks. The merge runs through t25 merge or the API, and records who triggered it. src/cli.ts, src/server/app.ts, merge in src/t25/service.ts

The reviewer's verdict does not get the last word either. T25 recomputes the verdict from the findings, and an in-scope finding blocks even if the model wrote APPROVE. The full story is in the post about the model's APPROVE.

What this does not solve

The checks show that the right path was followed, not that the code is correct. A small diff can hide a business-rule bug that no check catches, and a human who approves only the summary can let it through. There is also no per-layer review. sensitive_paths is binary: it marks a path as sensitive or not, and it only adds a human. Reviewing the outer edge of the architecture with fewer people is an idea on the roadmap, with no date.

For the diff ceiling itself, see the post on diff limits. The full settings are in the configuration reference.

How to apply this without T25

  1. Put a human at the merge. Automate everything before it, not after.
  2. Set the approval gate by the task's risk, not by team habit.
  3. Set a diff ceiling before the agent runs, and treat an overrun as a failure, not a warning.
  4. Compare approval against the PR's proof (checks, digest, identifiers), not against the agent's sentence.

FAQ

Does the factory need a human on every PR?

The merge is human by default. Risk changes how much of the work goes through approval before the build: plan, spec and PR. For low-risk tasks, the plan does not need approval.

Does T25 merge by itself when the review passes?

No. An approved review takes the task to the PR. The merge takes a command or an API call, records who ran it, and happens only after the repository's required checks pass.

Is lights-off always a bad idea?

No. The CRBugBench paper recommends dropping supervision only for narrow, low-consequence changes, calibrated on the repository itself. The mistake is treating broad automation as the default without measuring what it lets through.

Can I use the idea without T25?

Yes. The four steps above do not depend on the tool. T25 adds that the gate, the diff ceiling and the merge check already live in the pipeline, with the decision and the actor recorded.

Run T25 on your own machine.

Access is by invite: a personal download link arrives by e-mail, the installer verifies the package checksum and doctor --evaluation validates the environment. Free for the 30 days of the early evaluators program. The orchestrator, database and worktrees run on your machine; T25 does not collect your code, prompts, logs or credentials. Agent CLIs send data to their own providers, under your account and subject to their policies.

t25 doctor --evaluation
t25 create "Add retry with backoff to the HTTP client"

Request an invite

Semantic decisions in agent pipelines, with Jev

Semantic decisions in agent pipelines: T25 asks Jev for a typed signal, logs every answer, and lets deterministic policy decide the verdict and route.