Semantic decisions in agent pipelines, with Jev
T25 asks Jev, from TypeSafe, and lets a deterministic policy decide what the probability means.
When QA closes a criterion, T25 does not ask for a paragraph about the proof. It asks evidence.relation_to_criterion.v1: does the evidence support the criterion, contradict it, or say nothing. The answer is a Choice with three probabilities. That contract is what we mean by semantic decisions in agent pipelines.
Semantic decisions in agent pipelines: what we ask
The catalog lives in src/decisions/questions.ts. Every id has a version (v1) and a type. Jev, from TypeSafe, is a System One model: one POST to https://api.typesafe.ai/v1/systemone sends a state and several questions, and the typed answers come back together. The public introduction names three primitives. A Noul is the probability the answer is yes, from 0 to 1. A Choice picks from the list we sent and returns the distribution. A Score places the state on a rubric and also returns confidence.
Five families cover the judgment a conditional does not close:
| Family | Question | Type | What is at stake |
|---|---|---|---|
| Context | context.candidate_helps.v1 |
Noul | Does this file help the criteria, or does it only pad the prompt? |
| Progress | progress.looks_like_workaround.v1 |
Noul | Does the change fix the product, or does it weaken the test? |
| Evidence | evidence.relation_to_criterion.v1 |
Choice | Does the proof support the criterion, contradict it, or say nothing? |
| Attention | attention.finding_lacks_context.v1 |
Noul | Does the finding name a criterion or a file, or does someone have to open the rest? |
| Routing | routing.adapter.v1 |
Choice | Which CLI on the allowlist should try this role? |
QA fires evidence.relation_to_criterion.v1 once per criterion (supports, contradicts, says_nothing). On a stage change, progress sends four Nouls: whether the diff still attacks the open criterion, whether the change looks like a test workaround, whether the error fingerprint repeats, and whether QA describes the same gap. Context looks at the first eight paths in relatedFiles at most. With more than three skills, context.skill_candidate_helps.v1 is included.
quality.action.v1 and quality.scope.v1 run after the fact. decideQualityAction has already chosen rework, a QA rerun, or a person. The code's choice is stored in heuristic with probability 1 on the action taken and 0 on the others, and that Choice is not part of the state.
Why a typed probability audits better than free text
Free text carries the evidence and the conclusion in the same block. Review already showed us the cost: the model writes APPROVE and, a few lines later, lists an in-scope finding. evaluateReview() decides, by reading the findings again. That case is written up in The model's APPROVE is not the merge.
A System One answer has no second channel for a verdict to hide in. parseProviderAnswer(), in src/decisions/answers.ts, rejects anything outside the contract:
- A Noul outside
[0, 1]is dropped. - A Choice outside the enum we sent is dropped. So is a probability key that was not in the enum.
- The enum probabilities have to sum to 1, with a tolerance of
0.02. - A Score without a valid distribution is dropped. The stored
levelis the option with the highest probability, computed here.
Two rows with the same questionId and the same questionVersion sit on one axis. Changing the wording is a new version, not a silent edit of the history.
Shadow, active, and the fallback
The semantic_decisions section in the configuration starts off. Provider fake, mode: shadow, routing manual, model pinned to jev-1.13.0. The fake answers inside the process. The test suite does not call TypeSafe.
createSemanticDecisionEngine() computes the heuristic on every observation. With provider: jev and the required facts present, the call fits in timeout_ms (1500 ms). The clock covers the whole call, including up to two retries on HTTP 429 and 529. No key, a timeout, an unexpected status, or JSON the parser refuses: the row keeps a providerError, jev: null, and the heuristic. observeSemantic() catches the exception and the task continues.
A missing required fact never leaves the machine. projectObservationState() writes { absent: true }, sets factsStatus: insufficient, and skips Jev. In composeAttention(), a null answer or insufficient facts becomes unknown, and unknown asks for a person.
The YAML accepts mode: active. The stored row is still mode: "shadow" and policyAction: "observe". policyActionFor() returns nothing else, and assertNoControlFlow() rejects the rest. Promoting an action out of shadow requires assertPromotionAllowed(): a paired comparison, a holdout replay, human adjudication, incremental value, and a test that the path still runs without Jev.
The opt-in that can already change an execution choice is routing: automatic, with the layer on and provider jev. selectExperimentAdapter() switches CLI when routing.adapter.v1 reports confidence of at least 0.7 and the option is on the role's allowlist. Otherwise the first configured adapter stays, with fallbackReason of manual, incomplete_answers, low_confidence, not_authorized, or kept_default. The Score routing.role_difficulty.v1 only steps down inside the same CLI (two steps at trivial, one at easy, zero at moderate or hard), on the same floor. SKILL_SUGGESTION_PROTOCOL.ready is false, so the skill question can be logged while the prompt still gets the full list.
What each row stores
Each line of semantic-decisions.jsonl is a semantic-decision.v1: task, versioned questionId, provider, SHA-256 digest of the state, heuristic, answer or null, policyAction, error, responseModel, factsStatus, and absent keys. The log keeps the latest 4000 records. A line that fails to parse goes to semantic-decisions.invalid.jsonl.
On DONE, FAILED, or CANCELLED, stampLaterOutcome() fills an empty laterOutcome with accepted, failed, or cancelled. The original answer stays.
The state we send is a projection. send_code: true fails YAML loading; the only accepted value is false. The allowlist drops diff, patch, content, source, body, and prompt. A line that looks like source becomes [CODE_OMITTED]. Secrets go through the redactor. A payload that is still blocked is not asked. The key stays out of the YAML, in T25_JEV_API_KEY or in jev-secret.json mode 0600. t25 jev show and GET /api/v1/jev say whether a key exists. They never return the value.
The default model is jev-1.13.0. The public models page warns that the jev-latest alias starts pointing at another id when a release ships, and that the answers can move with it. A threshold tuned on one version stays pinned to it. responseModel stores the id that answered.
Where the threshold lives, and why we stop there
The public Noul docs put the threshold in the program and send the middle value to a person. noulBand() does that: 0.7 or above is positive, 0.3 or below is negative, and the middle is uncertain. composeAttention() reads how the evidence relates to the criterion, whether the finding has context, and whether the fingerprint repeats a project incident. One uncertain answer returns unknown. A contradiction, an opaque finding, or a repeated incident returns needs_human. skip_human comes out only when the whole set is decisive and none of those signals appear. It does not merge.
The merge to main stays with the person who answers for the repository. The review verdict stays in evaluateReview(). A spec becomes a plan only after a person approves it. The factory runs on the machine of the person who installed it and calls the agent CLIs that person already pays for. While a promotion proof is missing, the heuristic is the decision.
How to use the same contract in another pipeline
- One question per judgment, with the enum your code already knows how to handle.
- Send only the keys that question lists. The repository stays where it is.
- Fail closed on parse. An invalid body is an error, stored apart from the valid log.
- Persist the heuristic and the probability. Do not put the heuristic back into the
state. - Write the threshold in code, with a middle band that calls a person. Blow the deadline and continue on the heuristic.
FAQ
Does Jev approve the pull request or perform the merge?
The answer comes in as a signal. Review goes through evaluateReview(), which recomputes the verdict from the findings. The merge to main stays a human action, after the repository's required checks.
What happens if Jev expires in the middle of a task?
The heuristic remains. The row gets a providerError and jev: null. The task continues. On the attention path, a missing answer counts as unknown and asks for a person.
Why not ask for a verdict in prose and parse it?
Prose bundles the proof and the conclusion, and the parser becomes another place for the conclusion to hide. Choice, Score, and Noul return a value in the enum we defined, plus the distribution. We can reject the row and move the threshold without rereading a paragraph.
Can T25 run without a Jev key?
Yes. The default is the layer off and provider: fake. t25 jev show reports the provider, the mode, and whether a key exists. Pointing the provider at jev is opt-in, through the CLI, through PUT /api/v1/jev as an admin, or through the YAML, and the starting mode is still shadow.