Evaluation

API reference

Evaluation protocol for detector studies.

The package holds the pieces a published detector number is built from: group holdout that keeps every instance of a group on one side of the split (group_holdout, GroupSplit, GroupLeakage), ranking metrics that report a one-class fold as a refusal (ranking_metrics), the clock control that is published beside every detector (clock_control), and the worst-baseline alarm rule with the false-alarm floor it carries (worst_baseline_threshold, fires, far_floor, over_floor), and the conformal test martingale alarm with its record-level false-alarm bound (conformal_p_values, power_martingale, mixture_martingale, martingale_alarm).

Name	Summary
tsdive.eval.clock_control	Score every window by how far into its own record it sits.
tsdive.eval.conformal_p_values	Conformal p-value of each stream score against its pool, in stream order.
tsdive.eval.far_floor	False-alarm rate of a score that reads nothing, at k baseline windows.
tsdive.eval.fires	True when score is strictly above threshold.
tsdive.eval.group_holdout	Assign whole groups to n_folds folds, balancing strata.
tsdive.eval.martingale_alarm	First index where the log martingale reaches log(1 / delta), else None.
tsdive.eval.mixture_martingale	Natural log of the mean over epsilons of the power martingales.
tsdive.eval.over_floor	How far a measured false-alarm rate sits above far_floor(k_baseline).
tsdive.eval.power_martingale	Natural log of the power martingale prod(epsilon * p ** (epsilon - 1)).
tsdive.eval.ranking_metrics	Score a fold: ROC-AUC, PR-AUC and precision at a recall floor.
tsdive.eval.worst_baseline_threshold	The alarm threshold: the largest score the tool gave a baseline window.
tsdive.eval.GroupLeakage	A holdout group was found on both sides of a split.
tsdive.eval.GroupSplit	A k-fold split in which every group sits entirely inside one fold.
tsdive.eval.RankingMetrics	Threshold-free metrics of a detector's scores against binary labels.