precrec (version 0.24.0)

metric_table: Get every evaluation metric at every cutoff

Description

The metric_table function returns one row per cutoff and one column per metric, in the manner of pROC::coords and the cutoff slots of a ROCR::performance object. It is the table counterpart of metric_curve(), which projects one metric against another.

Usage

metric_table(x, scores = NULL, labels = NULL, metrics = NULL, at = NULL, ...)

Value

The metric_table function returns a data frame with one row per cutoff per model per test dataset - or, when at is given, one row per threshold per model per test dataset - and the following columns.

modname	Model name
dsid	Test dataset ID
at	The threshold asked for, when at is given
rank	Number of instances called positive, 0 to n
normalized_rank	rank / n, the x axis of the basic metric plots
score	The cutoff itself, see below
label	1 if the instance at this rank is positive, -1 if it is negative
...	One column per metric, named as evalmod() names them

Arguments

x

An S3 object created by the mmdata() function, or a basic-metric object created by evalmod(mode = "basic"). The metric_table function ignores scores and labels when x is specified. These arguments are internally passed to the mmdata() function when x is unspecified. In that case, both scores and labels must be at least specified.

scores

A numeric dataset of predicted scores. It can be a vector, a matrix, an array, a data frame, or a list.

labels

A numeric, character, logical, or factor dataset of observed labels. It can be a vector, a matrix, an array, a data frame, or a list.

metrics

A character vector of the metrics to calculate in addition to the default set, or the string "all" for every metric precrec knows, exactly as evalmod() takes it. It must be unspecified when x is already a basic-metric object, which carries the metrics it was built with.

at

A numeric vector of thresholds to report, instead of every cutoff. Each one is looked up rather than recalculated: score >= the threshold calls some number of instances positive, and that count is a rank the table already has a row for. Use it to read the metrics of a threshold chosen somewhere else - on other data, from a requirement, or by hand.

...

These additional arguments are passed to evalmod(), and through it to mmdata(), when this function builds the basic metrics itself. beta, cost_fp, cost_fn, basic_ties, modnames, dsids and posclass are the useful ones here.

What a row means

Row rank = k is the cutoff that calls the top k instances positive, so score is the score of the instance at rank k and the rule the row stands for is score >= that value. The metrics on the row are the metrics of that rule.

The first row is rank = 0 - call nothing positive. There is no instance at that rank, so score and label are NA, and it is kept because it is a real operating point.

The two end rows are missing one cell each. Precision is TP / (TP + FP), so the row that calls nothing positive has no denominator for it, and NPV is TN / (TN + FN), so the row that calls everything positive has none for that. Both are NA, together with the metrics derived from them - the false discovery rate is 1 - precision, the false omission rate is 1 - NPV, and markedness is precision + NPV - 1, so it is missing at both ends. Every other metric on those rows is measured.

The long form as.data.frame(evalmod(mode = "basic")) returns carries a value in those cells instead, taken from the neighboring row. That is what anchors the precision-recall curve at recall 0, and it is the right value for a curve - but it is not a measurement of the rule the row stands for, and a precision of 1 beside a sensitivity of 0 reads as a perfect threshold. Plot from the long form; decide from this table.

Tied scores share one value of each metric rather than taking a value that depends on the order the ties arrived in. basic_ties of evalmod() controls that.

Reading a threshold you already have

at answers the question the rest of the table cannot: what are the metrics of this threshold. A threshold is rarely one of the observed scores, so it has no row of its own, but it always names one of the cutoffs - score >= it calls a certain number of instances positive, and that count is a rank. The row returned is that rank's row, so at is what was asked for and score is the observed cutoff realizing it, the smallest score still called positive.

A threshold above every score gives the rank = 0 row, where score is NA because nothing is called positive, and precision is NA because a rule that makes no positive predictions has none. One below every score gives the rank = n row, with no NPV for the same reason at the other end. Neither is an edge case here: a threshold chosen on one dataset lands outside the range of another routinely, which is most of why it is worth asking what it does there. A score of NA is never a positive prediction, matching the na_worst = TRUE default of the rest of the package.

Ranks are per test dataset, so one threshold lands on a different rank in each of them. That is the reason to ask by threshold: a threshold is what transfers between datasets, and a rank is not. It is what makes a cutoff chosen on held-out data measurable on the data it was kept from, which the Choose an operating point article works through: https://evalclass.github.io/precrec/articles/howto-operating-point.html.

Several test datasets

A cutoff belongs to the dataset it was read off, so the rows are per dataset and nothing is averaged across them - unlike the basic metric plots, which average by default. A basic-metric object passed as x therefore has to have been built with raw_curves = TRUE when it holds more than one dataset, since an object built without it keeps only the average.

See Also

evalmod() for calculating the metrics, metric_curve() for projecting one metric against another, and as.data.frame() for the long form the plots use.

Examples

##################################################
### Single model & single test dataset
###

## Load a dataset with 10 positives and 10 negatives
data(P10N10)

## Get every metric at every cutoff
tab <- metric_table(scores = P10N10$scores, labels = P10N10$labels)
head(tab)

## The cutoff that maximizes the F-score
tab[which.max(tab$fscore), c("rank", "score", "precision", "fscore")]


##################################################
### Add metrics beyond the default set
###

lifted <- metric_table(
  scores = P10N10$scores, labels = P10N10$labels,
  metrics = c("lift", "jaccard")
)
head(lifted[, c("rank", "score", "precision", "lift", "jaccard")])


##################################################
### Read the metrics of thresholds you already have
###

## 14 is a run of six tied scores, so the rank is the end of the run
metric_table(
  scores = P10N10$scores, labels = P10N10$labels,
  at = c(17, 14, 6)
)[, c("at", "rank", "score", "sensitivity", "precision")]


##################################################
### Reuse an object that is already calculated
###

points <- evalmod(
  scores = P10N10$scores, labels = P10N10$labels,
  mode = "basic"
)
head(metric_table(points))