precrec (version 0.24.0)

average_precision: Calculate the average precision

Description

The average_precision function returns the average precision of every precision-recall curve of an S3 object generated by evalmod(). Average precision summarizes a precision-recall curve as the precision at each cutoff weighted by the recall it gains over the cutoff before it.

Usage

average_precision(curves)
# S3 method for aucs
average_precision(curves)

Value

The average_precision function returns a data frame with the columns modnames, dsids, aps and baselines. Unlike auc() it has one row per model and dataset rather than one per curve, because average precision is defined on the precision-recall curve only.

baselines is the proportion of positives, which is what average precision is worth by chance - see Reading an area against its baseline in auc(). It moves with the class balance, so an average precision quoted without it does not say how good the classifier is.

Arguments

curves

An S3 object generated by evalmod().

How this differs from the area under the curve

Average precision is \(AP = \sum_i (r_i - r_{i-1}) p_i\), summed over the cutoffs from the second one on. Joining the raw precision-recall points with horizontal steps is not the same as measuring the area under the curve through them, and it is not the same number: the step estimator reads high wherever the two disagree, because linear or stepwise movement between precision-recall points overstates what a classifier can actually achieve there.

auc() reports the area under the curve evalmod() has interpolated the correct way, which is the reason this package exists, and is the number to prefer. Average precision is reported because several other packages report it under this name, and because the size of the gap between the two is itself worth seeing.

The sum starts at the second cutoff, so the precision of the empty prediction set - 0/0, which every tool defines differently - never enters it.

See Also

auc() for the areas under the ROC and precision-recall curves. prbe() for the precision-recall break-even point. evalmod() for generating S3 objects with performance evaluation metrics.

Examples

##################################################
### Single model & single test dataset
###
samps <- create_sim_samples(1, 50, 50, "good_er")
sscurves <- evalmod(scores = samps[["scores"]], labels = samps[["labels"]])
average_precision(sscurves)

## The area under the properly interpolated curve, for comparison
auc(sscurves)

##################################################
### Multiple models & multiple test datasets
###
samps <- create_sim_samples(4, 50, 50, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]], dsids = samps[["dsids"]]
)
mmcurves <- evalmod(mdat)
average_precision(mmcurves)