Jev confidence thresholds

Every Jev Patterns page suggests Jev confidence thresholds. None of them are sent to the model. TypeSafe Jev returns a distribution and, for Choice and Score, a confidence scalar. Your code decides whether 0.74 is good enough to auto-assign a ticket. That decision is the whole operational art. This guide is how to stop treating confidence as a synonym for “sounds sure.”

What confidence is

For Choice and Score, Jev returns a full distribution and a confidence scalar derived from how peaked that distribution is. A department Choice at 0.86 / 0.08 / 0.03 / … is peaked. A 0.34 / 0.31 / 0.28 split is not, even if the argmax is “billing.” Jev confidence thresholds should look at the scalar, and you should still log the vector.

Calibration is measured across groups of predictions. It does not guarantee that an individual answer is correct. That is why the threshold lives in your code, next to the business action, not in the question instructions. Instructions that say “only answer if sure” fight the primitive. Ask the question. Threshold the answer.

A default Jev confidence threshold

if (answer.confidence >= 0.75) {
  autoApply(answer.choice);
} else {
  queueForHuman(answer);
}

0.75 is a starting point, not a law. Lower it on low-cost actions (tag a ticket). Raise it before you page someone or move money. Put the number in config so product can change it without a new prompt. Jev Patterns repeats this rule because every borrowed LLM JSON snippet on the internet invents a "confidence": "high" string instead.

Noul is already a probability

Noul has no separate confidence field. The value is the probability the statement is true. A Noul of 0.5 means unsure, not “medium.” Do not map 0.5 onto a yellow SLA bucket. Map the ends:

• noul >= 0.80 — treat as yes, if the cost of a false positive is acceptable
• noul <= 0.20 — treat as no
• the middle — human review, or another question with more state

For safety-sensitive Nouls (self_harm in a moderation screen, legal escalation) the auto-yes threshold should be much lower: you are routing to a person, not generating advice. That is still a Jev confidence threshold. It just sits on a different primitive.

Score can sit between levels

A Score of 2.14 on a 0–3 rubric is not “round to 2.” It is a probability-weighted position. If paging fires at 2.5, 2.14 should not page. If you need a hard bucket, threshold the weighted value, or read the mass on the top level — do not stringify the number. Urgency scoring and RAG chunk relevance both depend on this.

Worked Jev confidence thresholds by job

• Tagging a ticket with a department the human can override: 0.70 is often enough. The cost of a miss is a drag-and-drop.
• Auto-refund under $20: require policy Noul ≥ 0.85 and an amount cap in code. Money is not a tagging error.
• PagerDuty from a Score: fire on weighted score ≥ 2.5, not on confidence alone. A peaked Score of 0.4 (calm) should never page.
• self_harm Noul: escalate to a human playbook at a low number. False positives are cheaper than false negatives. That is still a Jev confidence threshold; the sign of the error is flipped.

Write the numbers in a table next to the pattern name. Review them when jev-latest moves. Aliases change answers. Jev confidence thresholds that lived in a Slack message will drift. Put them in git.

Confidence cannot save you from missing context. If refund policy is not in state, policy_supports_refund is a guess against the training prior. Pass the excerpt. Pass the last tool result. Pass the three prior tickets. Then the distribution has something to be peaked about. Jev confidence thresholds on empty state are theater.

Pin jev-latest or a versioned id when a threshold is a ship gate. Aliases move. Official note is on TypeSafe’s models page. Worked examples live on every pattern page. Start with support ticket routing. Read Choice, Score, Noul if you still mix the primitives. Jev Patterns is not affiliated with TypeSafe AI; calibrate on your labels anyway.

Logging Jev confidence thresholds

Store the question key, the winning Choice or weighted Score, the full probability vector, the confidence scalar, the threshold that fired, and the human override if any. Without overrides you cannot calibrate. Jev confidence thresholds that never meet a label are décor. A weekly plot of override rate versus threshold is the product review. If overrides cluster on one key, fix criteria, not the global 0.75.

Studio fixtures will not teach you this. They are canned bars. Put Jev confidence thresholds into staging on real tickets, with a human still in the loop, before you auto-assign in production. That sentence is the whole guide in operational form.