Computes Vargha and Delaney's (2000) A, the probability that a randomly drawn observation from the first sample exceeds a randomly drawn observation from the second (with tied pairs counted as half), together with its asymptotic standard error from DeLong, DeLong, and Clarke-Pearson (1988) and a confidence interval on the population A. A is equivalent to the receiver-operating-characteristic area-under-the-curve (AUC) and to the common-language effect size (McGraw & Wong, 1992) under a continuous-response assumption, and is a robust, scale-free ordinal effect size that does not require equal variances or normality.
vargha_delaney_A(
x,
y = NULL,
data = NULL,
conf_level = 0.95,
ci_method = c("logit", "wald")
)A one-row data.frame with columns A (point
estimate), se (DeLong-DeLong-Clarke-Pearson standard error),
lower_limit and upper_limit (confidence limits at
conf_level), z_value and p_value (Wald test of
\(H_0\!: A = 0.5\), i.e., stochastic equality), n_1 and
n_2 (group sample sizes), and ci_method.
Either a numeric vector of observations from group 1, or a
two-sided formula of the form outcome ~ group (in which case
data must be supplied and the grouping variable must have exactly
two levels).
Numeric vector of observations from group 2. Ignored when x
is a formula.
Optional data frame containing the variables named in the
formula x.
Confidence coverage for a symmetric interval (default
0.95).
Either "logit" (default; Wald interval on the
logit of A with back-transformation, recommended for finite
samples; Newcombe, 2006b) or "wald" (untransformed Wald on the
original scale, clipped to [0, 1]).
Ken Kelley [email protected]
Definition. For independent samples \(X_1, \ldots, X_{n_1}\) and \(Y_1, \ldots, Y_{n_2}\), $$A = \Pr(X > Y) + \tfrac{1}{2}\,\Pr(X = Y).$$ Values of \(A = 0.5\) indicate stochastic equality; \(A > 0.5\) indicates that group 1 tends to score higher. Qualitative magnitude labels for A are not reported here, in keeping with the DMAR convention of reporting effect sizes as numbers with confidence intervals.
Sample estimate. Equivalent rank-based computation: $$\hat A = \frac{\bar R_X - (n_1 + 1)/2}{n_2},$$ where \(\bar R_X\) is the mean rank of the first sample in the pooled ranking with mid-ranks for ties (Vargha & Delaney, 2000, p. 109). Equivalently, \(\hat A = U / (n_1 n_2)\), with \(U\) the Mann-Whitney U-statistic counting \(X_i > Y_j\) (tied pairs at \(1/2\)).
Standard error. The function uses the DeLong-DeLong-Clarke-Pearson (1988) U-statistic variance estimator, which is unbiased under sampling from any joint distribution (no parametric or homoscedasticity assumption). Defining the placement components $$V_{10}(X_i) = \frac{1}{n_2}\sum_{j} \psi(X_i, Y_j), \qquad V_{01}(Y_j) = \frac{1}{n_1}\sum_{i} \psi(X_i, Y_j),$$ with \(\psi(x, y) = 1, \tfrac{1}{2}, 0\) as \(x > y, =, <\), the variance estimate is $$\widehat{\mathrm{Var}}(\hat A) = \frac{S^2_{10}}{n_1} + \frac{S^2_{01}}{n_2},$$ where \(S^2_{10}\) and \(S^2_{01}\) are the sample variances of the \(V_{10}\) and \(V_{01}\) placement components. This is identical to the (single-curve) DeLong AUC variance and is the standard nonparametric variance for the Mann-Whitney functional (Brunner & Munzel, 2000).
Confidence interval. ci_method = "logit" (the default)
constructs a Wald interval on \(\mathrm{logit}(A) = \log\{A/(1-A)\}\)
using the delta method standard error
\(\widehat{\mathrm{SE}}(\hat A)/\{\hat A(1 - \hat A)\}\) and back-
transforms with the inverse logit. Newcombe (2006a, 2006b) showed in
extensive coverage simulations that logit-Wald has notably better small-
sample coverage than untransformed Wald, while remaining simple and free
of iteration. ci_method = "wald" returns the untransformed Wald
interval, clipped to \([0, 1]\).
Brunner, E., & Munzel, U. (2000). The nonparametric Behrens-Fisher problem: Asymptotic theory and a small-sample approximation. Biometrical Journal, 42(1), 17--25. tools:::Rd_expr_doi("10.1002/(SICI)1521-4036(200001)42:1<17::aid-bimj17>3.0.CO;2-U")
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837--845.
Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29--36.
McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111(2), 361--365. tools:::Rd_expr_doi("10.1037/0033-2909.111.2.361")
Newcombe, R. G. (2006a). Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 1: General issues and tail-area-based methods. Statistics in Medicine, 25(4), 543--557. tools:::Rd_expr_doi("10.1002/sim.2323")
Newcombe, R. G. (2006b). Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 2: Asymptotic methods and evaluation. Statistics in Medicine, 25(4), 559--573. tools:::Rd_expr_doi("10.1002/sim.2324")
Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2), 101--132. tools:::Rd_expr_doi("10.3102/10769986025002101")
smd, ci_smd, cohen_kappa
# Two numeric vectors.
set.seed(113)
x <- rnorm(40, mean = 0.6)
y <- rnorm(40, mean = 0)
vargha_delaney_A(x, y)
# Formula interface on the pygmalion field experiment. The first factor
# level (Control) forms group 1, so A below 0.5 says a randomly drawn
# control child tends to score below a child from the bloomer group.
vargha_delaney_A(iq_8 ~ treatment, data = pygmalion)
# Wald (untransformed) interval rather than logit.
vargha_delaney_A(x, y, ci_method = "wald")
Run the code above in your browser using DataLab