← All papers
First page of Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

Pedro Carvalho Brom

cs.LG Oct 26, 2026 · v1 stat.ML
Parts of propositions on selective-prediction ceilings and dynamic gates are machine-checked in Lean 4 with Mathlib, with proof files in the repository.
Typed decision models return a probability distribution over options supplied by the caller, and software tends to act on that number as if it were a guarantee. We measure what it licenses, comparing the open-weight Laya, in five checkpoints, with Jev 1.13, available only through an API. A probability licenses a statement about groups of decisions, not about one decision, and a gate that promises an accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic. Across 190 exchangeable evaluations, a threshold chosen by point estimate breaks the promise in 147, Learn then Test keeps it in all of them, and a Beta lower-bound rule decides more items on average than Learn then Test in 128 and fewer in none, without its guarantee. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. Ordinary typing noise breaks a promise calibrated on clean text, and, in exploratory runs, training with noise raises the noisy accuracy with no loss on clean text. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and does not get there on LIAR2. Under drift, the preregistered claim for a Bayesian retraining loop is not supported: the loop avoids needless retraining where the change does not degrade the model, at a cost equivalent to recalibration alone, and re-qualifying the threshold in every window keeps the promise more often where it does. On the monthly series of M3, neither decider predicts the direction of the next value better than a constant answer. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but it does not select faithful explanations, and exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals. We release the frozen answers with a datasheet, a promise check and the code.

Typed decision models such as Laya and Jev return probability distributions over caller-supplied options, and software often treats those confidences as guarantees. The question is what such a confidence actually licenses, in particular whether a gate promising a given accuracy on the decisions it takes alone keeps that promise.

The study gives a formal account of confidence measures, calibration, gate error types, Bayes thresholds and Beta success-run bounds. Selected parts of the propositions are checked in Lean 4 with Mathlib (SelectiveCeiling.lean, DynamicGate.lean). Five Laya checkpoints are compared with Jev 1.13 on MASSIVE in Portuguese and English, then under typing noise, across 52 locales, in further domains and under drift. Point-estimate thresholds are compared with Learn then Test and Beta lower-bound rules.

A point-estimate threshold breaks the accuracy promise in 147 of 190 evaluations, while Learn then Test keeps it in all of them. Calibrated in English, Jev keeps a 95% promise in only 8 of 51 locales, and typing noise breaks promises calibrated on clean text. A fine-tuned Laya overtakes Jev on Portuguese MASSIVE. Confidence separates right from wrong answers with AUROC from 0.80 to 0.96, but it does not select faithful explanations.

SystemAccuracyΔ vs Jev (pp)
Jev 1.130.947–
Laya multilingual0.731-21.6
Laya typed-decisions0.739-20.8
Laya multilingual, FT0.973+2.6
Laya typed-decisions, FT0.955+0.8
Accuracy on 1,000 MASSIVE pt-PT report utterances (subset of Table 4)