tsdive.eval.ranking_metrics ¶

ranking_metrics(
    y, scores, *, recall_floor: float = 0.5
) -> RankingMetrics

Score a fold: ROC-AUC, PR-AUC and precision at a recall floor.

ROC-AUC is the Mann-Whitney statistic, a tie counting one half, which equals sklearn.metrics.roc_auc_score. PR-AUC is the step-wise average precision sklearn.metrics.average_precision_score returns. precision_at_recall is the largest precision on the curve at any recall of at least recall_floor. A fold with one class present returns the refusal and None for every metric.

Examples:

>>> from tsdive.eval import ranking_metrics
>>> m = ranking_metrics([0, 0, 1, 1], [0.1, 0.4, 0.35, 0.8])
>>> m.roc_auc, round(m.pr_auc, 4), m.precision_at_recall
(0.75, 0.8333, 1.0)
>>> ranking_metrics([1, 1], [0.2, 0.3]).refusal
'fold under test carries one class only; no ranking metric exists'