A Temporal Algebraic Framework for Data Stream Evaluation Under Delayed Labels and Distribution Shifts
Ronaldo C. Prati, Elaine Ribeiro de Faria
cs.LG
Nov 16, 2026 · v1
cs.AI cs.DB
TL;DR
Twenty-five statements about temporal confusion-matrix aggregation are checked in Lean 4 against Mathlib, with the running example's values recomputed; the formalization is in a public repository.
Abstract
In data streams, labels often arrive late or only for some instances, class prevalence drifts, and rare classes vanish from short windows. Standard evaluation then confuses a degraded decision boundary with a change in prevalence, and is biased when labels are delayed or selected by the model. We extend an algebraic framework that obtains pooled (micro) and per-class (macro) measures from one confusion matrix to streams, weighting the counts in time by exponential fading or a sliding window; the update costs O(m^2) per step, whatever the stream length, and each pooled measure stays a weighted combination of the per-class ones. Delayed and selective feedback enter as an observation mask: until its label arrives, an instance is counted with the model's calibrated class probabilities, and the bias of this interim estimate is driven by their calibration error. Audited conventions report no value for a class without verified instances. We separate prior shift from concept drift by monitoring prior-free coordinates (recall and specificity) and attribute a change in a measure to the two sources with a first-order Jacobian decomposition. On synthetic streams and four real datasets, a detector on these coordinates with a memory per class raises no excess false alarms under label shift, unlike the Jacobian alarm and error-rate detectors. The correction for delayed labels recovers recall in some real settings and is only partial for precision, specificity and F1. Twenty-five statements are checked in Lean 4, and the code is in the aggmetrics package.
Problem
In data streams, labels can arrive late or only for some instances, class priors drift, and rare classes may be absent from short windows. Standard evaluation then confuses changes in the decision boundary with prior shifts, and it is biased under delayed or selective labels.
Approach
Extends an indicator-matrix algebraic framework for micro and macro measures to streams through a temporal operator that weights confusion counts by exponential fading or a sliding window, at O(m^2) cost per update. Delayed and selective feedback are modeled as an observation mask, and unlabeled instances are counted using calibrated class probabilities. Prior shift is separated from concept drift by monitoring prior-free coordinates and by a first-order Jacobian decomposition. Twenty-five statements are machine-checked in Lean 4 with Mathlib.
Results
A per-class detector on prior-free coordinates raises no excess false alarms under label shift, while Jacobian-alarm and error-rate detectors do. The delayed-label correction recovers recall in some real settings but only partially corrects precision, specificity and F1. Code is in the aggmetrics package, and the Lean formalization is public.