Covariance of two paired ROC curves

This function computes the covariance between the AUC of two correlated (or paired) ROC curves.

multivariate, utilities, nonparametric, ROC
# S3 method for default
# S3 method for auc
cov(roc1, roc2, ...)
# S3 method for smooth.roc
cov(roc1, roc2, ...)
# S3 method for roc
cov(roc1, roc2, method=c("delong", "bootstrap", "obuchowski"),
  reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE, boot.return=FALSE,
  progress=getOption("pROCProgress")$name, parallel=FALSE, ...)
roc1, roc2

the two ROC curves on which to compute the covariance. Either “roc”, “auc” or “smooth.roc” objects (types can be mixed as long as the original ROC curve are paired).


the method to use, either “delong” or “bootstrap”. The first letter is sufficient. If omitted, the appropriate method is selected as explained in details.


if TRUE (default) and the “roc” objects contain an “auc” field, re-use these specifications for the test. See details.


for method="bootstrap" only: the number of bootstrap replicates or permutations. Default: 2000.


for method="bootstrap" only: should the bootstrap be stratified (same number of cases/controls in each replicate than in the original sample) or not. Default: TRUE.


if TRUE and method="bootstrap", also return the bootstrapped values. See the “Value” section for more details.


the name of progress bar to display. Typically “none”, “win”, “tk” or “text” (see the name argument to create_progress_bar for more information), but a list as returned by create_progress_bar is also accepted. See also the “Progress bars” section of this package's documentation.


if TRUE, the bootstrap is processed in parallel, using parallel backend provided by plyr (foreach).

further arguments passed to or from other methods, especially arguments for cov.roc when calling cov, cov.auc or cov.smooth.roc. Arguments for auc (if reuse.auc=FALSE) and txtProgressBar (only char and style) if applicable.


This function computes the covariance between the AUC of two correlated (or paired, according to the detection of are.paired) ROC curves. It is typically called with the two roc objects of interest. Two methods are available: “delong” and “bootstrap” (see “Computational details” section below).

The default is to use “delong” method except with partial AUC and smoothed curves where “bootstrap” is employed. Using “delong” for partial AUC and smoothed ROCs is not supported.

For smoothed ROC curves, smoothing is performed again at each bootstrap replicate with the parameters originally provided. If a density smoothing was performed with user-provided density.cases or density.controls the bootstrap cannot be performed and an error is issued.

cov.default forces the usage of the cov function in the stats package, so that other code relying on cov should continue to function normally.


The numeric value of the covariance.

If boot.return=TRUE and method="bootstrap", an attribute resampled.values is set with the resampled (bootstrapped) values. It contains a matrix with the columns representing the two ROC curves, and the rows the boot.n bootstrap replicates.

AUC specification

To compute the covariance of the AUC of the ROC curves, cov needs a specification of the AUC. The specification is defined by:

  1. the “auc” field in the “roc” objects if reuse.auc is set to TRUE (default)

  2. passing the specification to auc with … (arguments partial.auc, partial.auc.correct and partial.auc.focus). In this case, you must ensure either that the roc object do not contain an auc field (if you called roc with auc=FALSE), or set reuse.auc=FALSE.

If reuse.auc=FALSE the auc function will always be called with to determine the specification, even if the “roc” objects do contain an auc field.

As well if the “roc” objects do not contain an auc field, the auc function will always be called with to determine the specification.

Warning: if the roc object passed to roc.test contains an auc field and reuse.auc=TRUE, auc is not called and arguments such as partial.auc are silently ignored.

Computation details

With method="bootstrap", the processing is done as follow:

  1. boot.n bootstrap replicates are drawn from the data. If boot.stratified is TRUE, each replicate contains exactly the same number of controls and cases than the original sample, otherwise if FALSE the numbers can vary.

  2. for each bootstrap replicate, the AUC of the two ROC curves are computed and stored.

  3. the variance (as per var.roc) of the resampled AUCs and their covariance are assessed in a single bootstrap pass.

  4. The following formula is used to compute the final covariance: \(Var[AUC1] + Var[AUC2] - 2cov[AUC1,AUC2]\)

With method="delong", the processing is done as described in Hanley and Hajian-Tilaki (1997) using the algorithm by Sun and Xu (2014).

With method="obuchowski", the processing is done as described in Obuchowski and McClish (1997), Table 1 and Equation 5, p. 1531. The computation of \(g\) for partial area under the ROC curve is modified as:

$$expr1 * (2 * pi * expr2) ^ {(-1)} * (-expr4) - A * B * expr1 * (2 * pi * expr2^3) ^ {(-1/2)} * expr3$$.

Binormality assumption

The “obuchowski” method makes the assumption that the data is binormal. If the data shows a deviation from this assumption, it might help to normalize the data first (that is, before calling roc), for example with quantile normalization:

    norm.x <- qnorm(rank(x)/(length(x)+1))
    cov(roc(response, norm.x, ...), ...)

“delong” and “bootstrap” methods make no such assumption.


If density.cases and density.controls were provided for smoothing, the error “Cannot compute the covariance on ROC curves smoothed with density.controls and density.cases.” is issued.


If “auc” specifications are different in both roc objects, the warning “Different AUC specifications in the ROC curves. Enforcing the inconsistency, but unexpected results may be produced.” is issued. Unexpected results may be produced.

If one or both ROC curves are “smooth.roc” objects with different smoothing specifications, the warning “Different smoothing parameters in the ROC curves. Enforcing the inconsistency, but unexpected results may be produced.” is issued. This warning can be benign, especially if ROC curves were generated with roc(…, smooth=TRUE) with different arguments to other functions (such as plot), or if you really want to compare two ROC curves smoothed differently.

If method="delong" and the AUC specification specifies a partial AUC, the warning “Using DeLong for partial AUC is not supported. Using bootstrap test instead.” is issued. The method argument is ignored and “bootstrap” is used instead.

If method="delong" and the ROC curve is smoothed, the warning “Using DeLong for smoothed ROCs is not supported. Using bootstrap instead.” is issued. The method argument is ignored and “bootstrap” is used instead.

DeLong ignores the direction of the ROC curve so that if two ROC curves have a different direction, the warning “"DeLong should not be applied to ROC curves with a different direction."” is printed. However, the spurious computation is enforced.

If boot.stratified=FALSE and the sample has a large imbalance between cases and controls, it could happen that one or more of the replicates contains no case or control observation, or that there are not enough points for smoothing, producing a NA area. The warning “NA value(s) produced during bootstrap were ignored.” will be issued and the observation will be ignored. If you have a large imbalance in your sample, it could be safer to keep boot.stratified=TRUE.

When both ROC curves have an auc of 1 (or 100%), their covariance will always be null. This is true for both “delong” and “bootstrap” and methods. This result is misleading, as the covariance is of course not null. A warning will be displayed to inform of this condition, and of the misleading output.


The covariance can only be computed on paired data. This assumption is enforced by are.paired. If the ROC curves are not paired, the covariance is 0 and the message “ROC curves are unpaired.” is printed. If your ROC curves are paired, make sure they fit are.paired criteria.


Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) ``Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach''. Biometrics 44, 837--845.

James A. Hanley and Karim O. Hajian-Tilaki (1997) ``Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: An update''. Academic Radiology 4, 49--58. DOI: 10.1016/S1076-6332(97)80161-4.

Nancy A. Obuchowski, Donna K. McClish (1997). ``Sample size determination for diagnostic accurary studies involving binormal ROC curve indices''. Statistics in Medicine, 16(13), 1529--1542. DOI: (SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.

Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves''. IEEE Signal Processing Letters, 21, 1389--1393. DOI: 10.1109/LSP.2014.2337313.

Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. Journal of Statistical Software, 40, 1--29. URL:

See Also

roc, var.roc

CRAN package plyr, employed in this function.

  • cov
  • cov.default
  • cov.auc
  • cov.smooth.roc
  • cov.roc

# Basic example with 2 roc objects
roc1 <- roc(aSAH$outcome, aSAH$s100b)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
cov(roc1, roc2)

# }
# The latter used Delong. To use bootstrap:
cov(roc1, roc2, method="bootstrap")
# Decrease boot.n for a faster execution:
cov(roc1, roc2, method="bootstrap", boot.n=1000)
# }
# To use Obuchowski:
cov(roc1, roc2, method="obuchowski")

# }
# Comparison can be done on smoothed ROCs
# Smoothing is re-done at each iteration, and execution is slow
cov(smooth(roc1), smooth(roc2))
# }
# or from an AUC (no smoothing)
cov(auc(roc1), roc2)

# }
# With bootstrap and return.values, one can compute the variances of the
# ROC curves in one single bootstrap run:
cov.rocs <- cov(roc1, roc2, method="bootstrap", boot.return=TRUE)
# var(roc1):
var(attr(cov.rocs, "resampled.values")[,1])
# var(roc2):
var(attr(cov.rocs, "resampled.values")[,2])
# }
# }
# Covariance of partial AUC:
roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")
cov(roc3, roc4)
# This is strictly equivalent to:
cov(roc3, roc4, method="bootstrap")

# Alternatively, we could re-use roc1 and roc2 to get the same result:
cov(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")
# }
# Spurious use of DeLong's test with different direction:
roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")
roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")
cov(roc5, roc6, method="delong")

## Test data from Hanley and Hajian-Tilaki, 1997
disease.present <- c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No",
                     "No", "Yes", "No", "No", "Yes", "No", "No")
field.strength.1 <- c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)
field.strength.2 <- c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)
roc7 <- roc(disease.present, field.strength.1)
roc8 <- roc(disease.present, field.strength.2)
# Assess the covariance:
cov(roc7, roc8)

# }
# With bootstrap:
cov(roc7, roc8, method="bootstrap")
# }
# }
Documentation reproduced from package pROC, version 1.16.2, License: GPL (>= 3)

Community examples

Looks like there are no examples yet.