usefun (version 0.5.2)

pr.test: Compare two Precision-Recall curves

Description

Test the hypothesis that the true difference in PR AUCs is equal to 0. We implement the same bootstrap method based on the idea from pROC::roc.test(). The PR AUC is calculated using PRROC::pr.curve() with the interpolation method of Davis (2006).

Usage

pr.test(
  labels,
  pred1,
  pred2,
  boot.n = 10000,
  boot.stratified = TRUE,
  alternative = "two.sided"
)

Value

a list with the AUCs of the two original prediction vectors and the p-value of the bootstrap-based test.

Arguments

labels

(numeric())
Vector of responses/labels (only two classes/values allowed: cases/positive class = 1 and controls/negative class = 0)

pred1

(numeric())
Vector of prediction values. Higher values denote positive class.

pred2

(numeric())
Vector of prediction values. Higher values denote positive class. Must have the same length as pred1.

boot.n

(numeric(1))
Number of bootstrap resamples. Default: 10000

boot.stratified

(logical(1))
Whether the bootstrap resampling is stratified (same number of cases/controls in each replicate as in the original sample) or not. It is advised to use stratified resampling when classes from labels are imbalanced. Default: TRUE.

alternative

(character(1))
Specifies the alternative hypothesis. Either "two.sided", "less" or "greater". Default: "two.sided".

References

Davis J, Goadrich M (2006). “The relationship between precision-recall and ROC curves.” Proceedings of the 23rd International Conference on Machine Learning, 148(4), 233--240. tools:::Rd_expr_doi("10.1145/1143844.1143874").

Examples

set.seed(42)
# imbalanced labels
labels = sample(c(0,1), 20, replace = TRUE, prob = c(0.8,0.2))
# predictions
pred1 = rnorm(20)
pred2 = rnorm(20)
pr.test(labels, pred1, pred2, boot.n = 1000, boot.stratified = FALSE)
pr.test(labels, pred1, pred2, boot.n = 1000, boot.stratified = TRUE)