precrec (version 0.24.0)

best_cutoff: Choose an operating point

Description

The best_cutoff function picks the cutoff that optimizes one evaluation metric, in the manner of pROC::coords(x, "best"). It returns one row per model per test dataset - a row of metric_table(), together with the criterion that chose it.

Usage

best_cutoff(x, scores = NULL, labels = NULL, metric = "youden", ...)

Value

The best_cutoff function returns a data frame with one row per model per test dataset, and the following columns.

modname	Model name
dsid	Test dataset ID
metric	The metric that was optimized
value	Its value at the chosen cutoff
rank	Number of instances called positive, 1 to n
normalized_rank	rank / n
score	The cutoff itself, the rule being score >= it
label	1 if the instance at this rank is positive, -1 if it is negative
...	One column per metric, as metric_table() returns them

The row is a row of metric_table(), so every other metric is there to be read at the same cutoff - which is the point of returning the whole row rather than the threshold alone.

Arguments

x

An S3 object created by the mmdata() function, or a basic-metric object created by evalmod(mode = "basic"). The best_cutoff function ignores scores and labels when x is specified. These arguments are internally passed to the mmdata() function when x is unspecified. In that case, both scores and labels must be at least specified.

scores

A numeric dataset of predicted scores. It can be a vector, a matrix, an array, a data frame, or a list.

labels

A numeric, character, logical, or factor dataset of observed labels. It can be a vector, a matrix, an array, a data frame, or a list.

metric

A string naming the metric to optimize. It can be any of the metrics evalmod() calculates, together with two names that come from the cutpoint literature rather than from the metric table.

Name	Metric	Optimum
"youden"	informedness	Maximum
"topleft"	roc_dist	Minimum
"fscore"	fscore	Maximum
"mcc"	mcc	Maximum
"cost"	cost	Minimum

The direction is a property of the metric, so it is not an argument - error, cost, roc_dist, the two error rates and the negative likelihood ratio are minimized, and the rest are maximized. score, label, predicted_positive_rate and predicted_negative_rate describe a cutoff rather than score it, and are refused.

...

These additional arguments are passed to evalmod(), and through it to mmdata(), when this function builds the metrics itself. beta for "fscore", cost_fp and cost_fn for "cost", and basic_ties, modnames, dsids and posclass are the useful ones here.

The criterion matters more than the threshold

Youden's J and the closest point to the top left corner are the two criteria most often reached for, and both are computed from sensitivity and specificity alone. Both of those are conditioned on the true class, so neither knows the prevalence, and on imbalanced data the cutoff they choose can sit at a precision no one would deploy. That is the argument this package makes about the ROC curve, one step further down the pipeline.

Prevalence-blind	"youden", "topleft"
Prevalence-aware	"fscore", "mcc", "cost"

The default is "youden" because that is what a caller arriving from another package expects. On imbalanced data it is the wrong default, and "mcc" or a "cost" weighted by what the two mistakes actually cost is the better choice. The Balanced and imbalanced data article shows the two disagreeing on the same dataset: https://evalclass.github.io/precrec/articles/howto-imbalanced-data.html.

Ties, and what is not here

Several cutoffs can share the optimum. The one with the smallest rank is returned - the threshold that calls the fewest instances positive - so the result is one row per model per test dataset whatever the data does.

Rows that share a score are an exception, because they are one threshold seen several times rather than several cutoffs: score >= that value calls all of the tied instances positive, so the rank reported is the last of the run and not the first. That is what evalmod(basic_ties = "hold") produces throughout, since it gives every cutoff in a run of tied scores the counts of the whole run.

The rank 0 row of metric_table() - the rule that calls nothing positive - is not a candidate. There is no threshold that predicts nothing, so its score is NA and it is not an operating point, which is what this function returns. It would otherwise win outright wherever the empty rule is optimal, and win on the tie-break wherever it merely ties: precision at rank 0 is the limit from above, so it is 1 whenever the top-ranked instance is a positive, and any classifier that ranks one first would come back with no threshold at all.

A metric that has no interior optimum is still optimized at an end of the range, and no warning says so: sensitivity is largest when everything is called positive, and specificity when as little as possible is. Those are the correct answers to the question asked, and rarely the question meant. accuracy and error have an interior optimum on balanced data and lose it as positives get rare, which is the same trap arriving by a different route: at a few percent positives, calling almost nothing positive is close to the most accurate thing a classifier can do.

A cutoff chosen on the same data the model is evaluated on is optimistic, by however much the criterion was free to chase. Choosing it on held-out data is the fix, and it is the caller's to make: nothing here resamples or cross-validates the choice.

See Also

metric_table() for every metric at every cutoff, which this function takes one row of, evalmod() for calculating the metrics, and prbe() for the precision-recall break-even point, which is an operating point defined by a crossing rather than by an optimum.

Examples

##################################################
### Single model & single test dataset
###

## Load a dataset with 10 positives and 10 negatives
data(P10N10)

## The cutoff that maximizes Youden's J
best_cutoff(scores = P10N10$scores, labels = P10N10$labels)


##################################################
### A criterion that knows the prevalence
###

best_cutoff(scores = P10N10$scores, labels = P10N10$labels, metric = "mcc")

## Weighted by what the two mistakes cost
best_cutoff(
  scores = P10N10$scores, labels = P10N10$labels,
  metric = "cost", cost_fp = 1, cost_fn = 5
)


##################################################
### Multiple models & multiple test datasets
###

samps <- create_sim_samples(2, 50, 50, c("poor_er", "good_er"))
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]], dsids = samps[["dsids"]]
)
best_cutoff(mdat, metric = "fscore")[, 1:7]