precrec (version 0.24.0)

auc_ci: Calculate CIs of ROC and precision-recall AUCs

Description

The auc_ci function takes an S3 object generated by evalmod() and calculates CIs of AUCs when multiple data sets are specified.

Usage

auc_ci(curves, alpha = NULL, dtype = NULL)
# S3 method for aucs
auc_ci(curves, alpha = 0.05, dtype = "normal")
# S3 method for aucboot
auc_ci(curves, alpha = 0.05, dtype = NULL)
# S3 method for aucdelong
auc_ci(curves, alpha = 0.05, dtype = NULL)

Value

The auc_ci function returns a dataframe of AUC CIs, with a baselines column beside the area giving what that area would be by chance - 0.5 for a ROC curve and the proportion of positives for a precision-recall curve. See Reading an area against its baseline in auc().

Over several test datasets the baseline is averaged over the same datasets the mean area is, so a fold that could not be evaluated is left out of both. From auc_boot() or auc_delong() it is the balance of the single test set.

An interval is the point of this function: the prevalence is what an area is worth by chance in the limit, and an area from a finite sample scatters around it, so an area sitting above its baseline means little on its own. Read the baseline against the interval instead - see The baseline is an asymptote in auc().

Arguments

curves

An S3 object generated by evalmod(). The auc_ci function accepts the following S3 objects.

S3 object	# of models	# of test datasets
smcurves	single	multiple
mmcurves	multiple	multiple

See the Value section of evalmod() for more details.

It also accepts the two objects that describe the uncertainty of a single test set: an aucboot object from auc_boot(), which gives a percentile interval, and an aucdelong object from auc_delong(), which gives a normal interval around DeLong's analytic standard error.

alpha

A numeric value of the significant level (default: 0.05)

dtype

A string to specify the distribution used for CI calculation.

dtype	distribution
normal (default)	Normal distribution
z	Normal distribution
t	t-distribution

See Also

evalmod() for generating S3 objects with performance evaluation metrics. auc() for retrieving a dataset of AUCs. auc_boot() and auc_delong() for a single test set.

Examples

##################################################
### Single model & multiple test datasets
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]],
  dsids = samps[["dsids"]]
)

## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat)

## Calculate CI of AUCs
sm_auc_cis <- auc_ci(smcurves)

## Shows the result
sm_auc_cis

##################################################
### Multiple models & multiple test datasets
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]],
  dsids = samps[["dsids"]]
)

## Generate an mscurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat)

## Calculate CI of AUCs
mm_auc_ci <- auc_ci(mmcurves)

## Shows the result
mm_auc_ci