pROC (version 1.3.1)

roc.test: Compare the AUC of two correlated ROC curves

Description

This function compares the AUC or partial AUC of two correlated (or paired) ROC curves. Several syntaxes are available: two object of class roc (which must have the same response data, but can be AUC or smoothed ROC), or either three vectors (response, predictor1, predictor2) or a response vector and a matrix or data.frame with two columns (predictors).

Usage

roc.test(...)
## S3 method for class 'roc':
roc.test(roc1, roc2, method=c("delong", "bootstrap",
"venkatraman"), alternative = c("two.sided", "less", "greater"),
reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE, ties.method="first",
progress=getOption("pROCProgress")$name, ...)
## S3 method for class 'auc':
roc.test(roc1, roc2, ...)
## S3 method for class 'smooth.roc':
roc.test(roc1, roc2, ...)
## S3 method for class 'formula':
roc.test(formula, data, ...)
## S3 method for class 'default':
roc.test(response, predictor1, predictor2=NULL,
na.rm=TRUE, method=NULL, ...)

Arguments

roc1, roc2

the two ROC curves to compare. Either roc, auc or smooth.roc objects (types can be mixed as long as

response

a vector or factor, as for the roc function.

predictor1

a numeric or ordered vector as for the roc function, or a matrix or data.frame with predictors two colums.

predictor2

only if predictor1 was a vector, the second predictor as a numeric vector.

formula

a formula of the type response~predictor1+predictor2.

data

a matrix or data.frame containing the variables in the formula. See model.frame for more details.

na.rm

if TRUE, the observations with NA values will be removed.

method

the method to use, either delong, bootstrap or venkatraman. The first letter is sufficient. If omitted, the appropriate method is selected as explained in details.

alternative

specifies the alternative hypothesis. Either of two.sided, less or greater. The first letter is sufficient. Default: two.sided. Only two.sided is available wi

reuse.auc

if TRUE (default) and the roc objects contain an auc field, re-use these specifications for the test. See details.

boot.n

for method="bootstrap" and method="venkatraman" only: the number of bootstrap replicates or permutations. Default: 2000.

boot.stratified

for method="bootstrap" only: should the bootstrap be stratified (same number of cases/controls in each replicate than in the original sample) or not. Ignored with method="venkatraman". Default: TRUE.

ties.method

for method="venkatraman" only: argument for rank specifying how ties are handled. Defaults to first as described in the paper.

progress

the name of progress bar to display. Typically none, win, tk or text (see the name argument to create_progress_b

...

further arguments passed to or from other methods, especially arguments for roc and roc.test.roc when calling roc.test.default or roc.test.formula. Arguments f

Value

• A list of class "htest" with following content:
• p.value
  the p-value of the test.
• statistic
  the value of the Z (method="delong") or D (method="bootstrap") statistics.
• alternative
  the alternative hypothesis.
• method
  the character string DeLong's test for two correlated ROC curves (if method="delong") or Bootstrap test for two correlated ROC curves (if method="bootstrap").
• null.value
  the expected value of the statistic under the null hypothesis, that is 0.
• estimate
  the AUC in the two ROC curves.
• data.name
  the names of the data that was used.
• parameter
  for method="bootstrap" only: the values of the boot.n and boot.stratified arguments.

encoding

UTF-8

AUC specification

The comparison of the AUC of the ROC curves needs a specification of the AUC. The specification is defined by:

1. theaucfield in therocobjects ifreuse.aucis set toTRUE(default)
2. passing the specification toaucwith...(argumentspartial.auc,partial.auc.correctandpartial.auc.focus). In this case, you must ensure either that therocobject do not contain anaucfield (if you calledrocwithauc=FALSE), or setreuse.auc=FALSE.

If reuse.auc=FALSE the auc function will always be called with ... to determine the specification, even if the roc objects do contain an auc field.

As well if the roc objects do not contain an auc field, the auc function will always be called with ... to determine the specification.

The AUC specification is ignored in the Venkatraman test.

Warning: if the roc object passed to roc.test contains an auc field and reuse.auc=TRUE, auc is not called and arguments such as partial.auc are silently ignored.

Computation details

With method="bootstrap", the processing is done as follow:

1. boot.nbootstrap replicates are drawn from the data. IfstratifiedisTRUE, each replicate contains exactly the same number of controls and cases than the original sample, otherwise ifFALSEthe numbers can vary.
2. for each bootstrap replicate, the AUC of the two ROC curves are computed and the difference is stored.
3. The following formula is used:$$D=\frac{AUC1-AUC2}{s}$$where s is the standard deviation of the bootstrap differences and AUC1 and AUC2 the AUC of the two (original) ROC curves.
4. Dis then compared to the normal distribution, according to the value ofalternative.

With method="delong", the processing is done as described in DeLong et al. (1988). Only comparison of two ROC curves is implemented.

With method="venkatraman", the processing is done as described in Venkatraman and Begg (1996) with boot.n permutation of sample ranks.

Warnings

If auc specifications are different in both roc objects, the warning Different AUC specifications in the ROC curves. Enforcing the inconsistency, but unexpected results may be produced. is issued. Unexpected results may be produced.

If method="delong" and the AUC specification specifies a partial AUC, the warning Using DeLong's test for partial AUC is not supported. Using bootstrap test instead. is issued. The method argument is ignored and bootstrap is used instead.

If method="delong" and the ROC curve is smoothed, the warning Using DeLong's test for smoothed ROCs is not supported. Using bootstrap test instead. is issued. The method argument is ignored and bootstrap is used instead.

If method="venkatraman", and the AUC specification specifies a partial AUC, the AUC specification is ignored with the warning Partial AUC is ignored in Venkatraman's test..

If method="venkatraman", and alternative is less or greater, the warning Only two-sided tests are available for Venkatraman. is produced and a two tailed test is performed instead.

Both DeLong and Venkatraman's test ignores the direction of the ROC curve so that if two ROC curves have a different differ in the value of direction, the warning (DeLong|Venkatraman)'s test should not be applied to ROC curves with a different direction. is printed. However, the spurious test is enforced.

If boot.stratified=FALSE and the sample has a large imbalance between cases and controls, it could happen that one or more of the replicates contains no case or control observation, or that there are not enough points for smoothing, producing a NA area. The warning NA value(s) produced during bootstrap were ignored. will be issued and the observation will be ignored. If you have a large imbalance in your sample, it could be safer to keep boot.stratified=TRUE.

Errors

Both Delong and Bootstrap tests work only on paired data. This assumption is enforced by the verification that the responses of roc1 and roc2 are identical. If they are found different, and the difference cannot be explained by missing values, the error The ROC test is defined only on paired ROC curves is produced.

An error will also occur if you give a predictor2 when predictor1 is a matrix or a data.frame, if predictor1 has more than two columns, or if you do not give a predictor2 when predictor1 is a vector. If density.cases and density.controls were provided for smoothing, the error Cannot compute the statistic on ROC curves smoothed with density.controls and density.cases. is issued.

With method="venkatraman", the processing is done as described in Venkatraman and Begg (1996) with boot.n permutation of sample ranks (with ties breaking). For consistency reasons, the same argument boot.n as in bootstrap defines the number of permutations to execute, even though no bootstrap is performed.

Acknowledgements

We would like to thank E. S. Venkatraman and Colin B. Begg for their support in the implementation of their test.

Details

This function compares two ROC curves. It is typically called with the two roc objects to compare. roc.test.default is provided as a convenience method and creates two roc objects before calling roc.test.roc.

Three methods are available: delong, bootstrap and venkatraman (see Computational details section below). delong and bootstrap are tests over the AUC whereas venkatraman compares the the ROC curves themselves.

Default is to use delong method except for comparison of partial AUC, smoothed curves and curves with different direction, where bootstrap is used. Using delong for partial AUC and smoothed ROCs is not supported in pROC (a warning is produced and bootstrap is employed instead). It is spurious to use delong for roc with different direction (a warning is issued but the spurious comparison is enforced). venkatraman's test cannot be employed to compare smoothed ROC curves. Additionally, partial AUC specifications are ignored (with a warning), and comparison of ROC curves with different direction should be used with care (a warning is produced as well).

If alternative="two.sided", a two-sided test for difference in AUC is performed. If alternative="less", the alternative is that the AUC of roc1 is smaller than the AUC of roc2. For method="venkatraman", only two.sided test is available.

For smoothed ROC curves, smoothing is performed again at each bootstrap replicate with the parameters originally provided. If a density smoothing was performed with user-provided density.cases or density.controls the bootstrap cannot be performed and an error is issued.

References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) ``Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach''. Biometrics 44, 837--845. James A. Hanley and Barbara J. McNeil (1982) ``The meaning and use of the area under a receiver operating characteristic (ROC) curve''. Radiology 143, 29--36. Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller (2009) ``Combination of protein biomarkers''. useR! 2009, Rennes. http://www.agrocampus-ouest.fr/math/useR-2009/abstracts/user_author.html

E. S. Venkatraman and Colin B. Begg (1996) ``A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment''. Biometrika 83, 835--848. DOI: 10.1093/biomet/83.4.835

See Also

roc

Examples

data(aSAH)

# Basic example with 2 roc objects
roc1 <- roc(aSAH$outcome, aSAH$s100b)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
roc.test(roc1, roc2)

# The latter used Delong's test. To use bootstrap test:
roc.test(roc1, roc2, method="bootstrap")
# Increase boot.n for a more precise p-value:
roc.test(roc1, roc2, method="bootstrap", boot.n=10000)

# Alternative syntaxes
roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns)
roc.test(aSAH$outcome, data.frame(aSAH$s100b, aSAH$wfns))

# If we had a good a priori reason to think that wfns gives a
# better classification than s100b (in other words, AUC of roc1
# should be lower than AUC of roc2):
roc.test(roc1, roc2, alternative="less")

# Comparison can be done on smoothed ROCs
# Smoothing is re-done at each iteration, and execution is slow
roc.test(smooth(roc1), smooth(roc2))
# or:
roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns, smooth=TRUE,
smooth.method="density", boot.n=100)
# or from an AUC (no smoothing)
roc.test(auc(roc1), roc2)

# Comparison of partial AUC:
roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc.test(roc3, roc4)
# This is strictly equivalent to:
roc.test(roc3, roc4, method="bootstrap")

# Alternatively, we could re-use roc1 and roc2 to get the same result:
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")

# Spurious use of DeLong's test with different direction:
roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")
roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")
roc.test(roc5, roc6, method="delong")

# Comparisons of the ROC curves
roc.test(roc1, roc2, method="venkatraman")