# Lights-on: why the software factory keeps a human at the merge

> Teams that tried to take people out of the code ended up reading it again. Here is what T25 keeps at merge, at the risk gate and at the diff ceiling.

- By: Djan Magno
- Published: 2026-10-13
- Updated: 2026-10-13
- URL: https://t25.io/en/blog/lights-on-human-at-merge/
- Engineering · software factory, human in the loop, AI code review, approval gate, lights-off, coding agents

> **TL;DR:** > - Lights-off is a software factory model where nobody reads the code. Teams that ran it for a few months say agents start stalling on problems that require understanding the code, because the reward is binary: the tests pass and nothing else breaks.
> - AI review catches a lot, but not everything. In CRBugBench's best case, a known bug shows up in 71.9% of tasks within the top five comments, and the paper reports weaker results on complex changes.
> - T25 is lights-on by design: a human starts the merge, plan approval depends on risk, and the diff has a configurable ceiling.
> - None of this rules out automation. It puts a human where a mistake is most expensive to undo.

## Three sources that point at the same limit

Dex Horthy, from HumanLayer, gave the talk [Why Software Factories Fail](https://youtu.be/Ib5GBkD555M). His core argument is that a well-built harness is not enough. His team tried lights-off, and by his account agents started struggling after three to six months, on problems nobody was reading the code for anymore. He blames the reward signal. The agent is rewarded when the problem is fixed and nothing breaks. That signal does not penalize poor design or the erosion of maintainability, and the cost only shows up months later. His answer is to align before coding: architecture, then program design (types, signatures, call stacks), then vertical slices. And keep reading the code.

pasquadev answered the talk in [a video](https://youtu.be/NY5Y2v-Z1tA) that describes how his team works: a PRD reviewed with product, a peer review with other staff engineers before any code, and pull requests of around 300 lines. He also uses hexagonal architecture as a yardstick. The deeper a diff reaches into the domain, the more it needs a human. Near the edges, less. He says he is still thinking this through and has not built it.

The [CRBugBench](https://arxiv.org/abs/2610.13668) paper (arXiv:2610.13668) measures automated reviewers on pull requests that introduced known bugs. Its metric, recall@k, is the fraction of tasks where the known bug appears among the first k findings. With Claude Opus 5 on Copilot CLI, the paper reports 71.9% at top-5. That figure averages two sets: 79.6% of bugs found before merge and 64.2% found after. The same analysis shows the detail that matters. 95.6% of runs inspect the defective file, but only 80.2% produce a finding for it, and 56.2% actually hit the bug. Finding the file is not the same as diagnosing the bug. The paper also says reviewers lose effectiveness on complex changes and on bugs that need reasoning beyond the diff. Its recommendation is to keep a human responsible for consequential changes and for maintainability.

Corey Gallon, in [another video](https://youtu.be/dYh7TQkuYFE), says roughly four of every ten sessions he reviewed contained an agent error that counted. His method is agent-based review in three levels that rise with the stakes. He admits he cannot check everything himself, so his design does not put a human on every step. That is where it differs from T25: here, the merge is not delegated.

## What T25 does today

| Control point | How it works in T25 | Where it lives |
|---|---|---|
| Spec | Human approval. No policy promotion removes this step. | `src/decisions/action-promotion.ts` |
| Plan | Risk-based: `low` does not require plan approval by default, `medium` does, `high` requires plan and PR, `critical` always requires a human. Each project can turn the gate on or off. | `src/core/config.ts`, `requiresPlanApproval` in `src/core/policy.ts` |
| Diff | A ceiling of 40 files, 2000 lines and 0.5 duplicate-line ratio, all configurable. Over the limit, the task fails and gets a handoff with the numbers. | `limits` in `src/core/config.ts`, `checkDiffLimits` in `src/core/policy.ts` |
| Sensitive paths | `review.sensitive_paths` per project. A PR that touches those paths proceeds only with a human acknowledgement, and the acknowledgement is bound to the diff digest: if the PR changes, it asks again. | `merge` in `src/t25/service.ts` |
| Merge | Human. The factory opens the PR and checks the required status checks. The merge runs through `t25 merge` or the API, and records who triggered it. | `src/cli.ts`, `src/server/app.ts`, `merge` in `src/t25/service.ts` |

The reviewer's verdict does not get the last word either. T25 recomputes the verdict from the findings, and an in-scope finding blocks even if the model wrote APPROVE. The full story is in the [post about the model's APPROVE](https://t25.io/en/blog/the-models-approve-is-not-the-merge/).

## What this does not solve

The checks show that the right path was followed, not that the code is correct. A small diff can hide a business-rule bug that no check catches, and a human who approves only the summary can let it through. There is also no per-layer review. `sensitive_paths` is binary: it marks a path as sensitive or not, and it only adds a human. Reviewing the outer edge of the architecture with fewer people is an idea on the roadmap, with no date.

For the diff ceiling itself, see the [post on diff limits](https://t25.io/en/blog/diff-limits-runaway-coding-agent/). The full settings are in the [configuration reference](https://t25.io/en/docs/).

## How to apply this without T25

1. Put a human at the merge. Automate everything before it, not after.
2. Set the approval gate by the task's risk, not by team habit.
3. Set a diff ceiling before the agent runs, and treat an overrun as a failure, not a warning.
4. Compare approval against the PR's proof (checks, digest, identifiers), not against the agent's sentence.

## FAQ

### Does the factory need a human on every PR?

The merge is human by default. Risk changes how much of the work goes through approval before the build: plan, spec and PR. For low-risk tasks, the plan does not need approval.

### Does T25 merge by itself when the review passes?

No. An approved review takes the task to the PR. The merge takes a command or an API call, records who ran it, and happens only after the repository's required checks pass.

### Is lights-off always a bad idea?

No. The CRBugBench paper recommends dropping supervision only for narrow, low-consequence changes, calibrated on the repository itself. The mistake is treating broad automation as the default without measuring what it lets through.

### Can I use the idea without T25?

Yes. The four steps above do not depend on the tool. T25 adds that the gate, the diff ceiling and the merge check already live in the pipeline, with the decision and the actor recorded.
