ci_eta_squared_generalized: Confidence Interval for Generalized Eta Squared (Approximate)

Description

Returns the point estimate of generalized eta squared ($\eta^2_G$; Olejnik & Algina, 2003) along with an optional confidence interval computed by one of two approximate methods. The default is to return only the point estimate (method = "none"), because both available CI methods are approximations whose coverage properties have not been broadly validated for this estimand and warrant independent evaluation before being used in substantive inference.

Usage

ci_eta_squared_generalized(
  object = NULL,
  observed = NULL,
  SS_effect = NULL,
  SS_observed = NULL,
  SS_error = NULL,
  F_effect = NULL,
  df_effect = NULL,
  F_observed = NULL,
  df_observed = NULL,
  df_error = NULL,
  N = NULL,
  method = c("none", "parametric", "bootstrap"),
  B = 10000L,
  conf_level = 0.95,
  alpha_lower = NULL,
  alpha_upper = NULL,
  seed = NULL
)

Value

A data.frame with one row per focal effect and the columns effect, eta_squared_generalized,

lower_limit, upper_limit, and method. For

aovlist fits a stratum column is also present. When

method = "none", the limit columns contain NA.

Arguments

object: Optional. A fitted model object of class aov, lm, or aovlist (multi-stratum aov fit for within-subjects / mixed designs).
observed: Character vector of factor names treated as measured.
SS_effect, SS_observed, SS_error: Sums of squares (option 2 in eta_squared_generalized).
F_effect, df_effect, F_observed, df_observed, df_error: F-values and degrees of freedom (option 3 in eta_squared_generalized).
N: Total sample size. Required when method = "parametric" and no fitted model is supplied; ignored when method = "none" or when a fitted model is supplied (derived automatically, via nobs(object) for single-stratum fits, or by summing degrees of freedom across error strata for aovlist).
method: One of "none" (default), "parametric", or "bootstrap". See Details.
B: Integer. Number of bootstrap replications when method = "bootstrap". Minimum 1000 (enforced); default 10000. We recommend 10000 or more for publication-quality intervals.
conf_level: Desired confidence coverage; default 0.95.
alpha_lower, alpha_upper: Optional Type I error on the lower and upper side.
seed: Optional integer seed for the bootstrap, for reproducibility. Used locally: the caller's random number generator state is restored on exit. Default NULL leaves the random number generator state alone.

Author

Ken Kelley [email protected]

Details

Why CI = "none" is the default. Confidence interval construction for $\eta^2_G$ is not as settled as for partial $\eta^2$ or $\omega^2$, because the denominator mixes sums of squares from heterogeneous sources (the focal effect, one or more measured factors, and the error term). No noncentral F transformation maps the population noncentrality parameter directly to $\eta^2_G$. Both methods below are approximations and are exposed for exploration rather than as defaults.

method = "parametric". The function first obtains a confidence interval for the population noncentrality parameter $\lambda$ of the focal effect's F-test via ci_nc_F. The NCP bounds are mapped through the partial-$\eta^2$ transformation $\eta^2_{p,\text{bound}} = \lambda_{\text{bound}}/(\lambda_{\text{bound}} + N)$ (matching the convention used by ci_pvaf and ci_omega_squared), and then re-expressed as $\eta^2_G$ bounds via $$\eta^2_{G,\text{bound}} = \frac{r_{\text{bound}}}{r_{\text{bound}} + r_{\text{obs}} + 1},$$ where $r_{\text{bound}} = \eta^2_{p,\text{bound}}/(1-\eta^2_{p,\text{bound}})$ and $r_{\text{obs}} = \sum \mathit{SS}_{\text{measured}}/\mathit{SS}_{\text{error}}$. This treats the observed-factor sums of squares as fixed at their sample values, so the interval inherits whatever sampling variability those contribute. It has not been validated for coverage and should be treated as preliminary.

method = "bootstrap". A residual bootstrap from the fitted model: the resampling unit is a residual, drawn nonparametrically from the model's own residuals rather than from a fitted distribution. For each of the B replications, the response is regenerated as $\hat{y}_i + \varepsilon^*_i$ where $\varepsilon^*$ is sampled with replacement from the model's residuals; the model is refit; $\eta^2_G$ is recomputed; and the percentile interval, the empirical quantiles of the B bootstrap values (Efron & Tibshirani, 1993), is reported. The percentile interval is the only bootstrap interval offered; there is no bias-corrected and accelerated (BCa) variant. Replicates whose refit fails are dropped, and the interval is computed from the replications that return a value. Bootstrap results vary from run to run; supply seed for reproducibility. Requires a single-stratum aov/lm fit. Not yet supported for aovlist (multi-stratum / within-subjects) fits, since the bootstrap needs to respect the subject-level correlation structure, which naive residual resampling does not. Use method = "parametric" for aovlist fits. Coverage has not been broadly validated for this estimand.

Within-subjects designs (aovlist). For multi-stratum fits the parametric CI uses each focal effect's stratum-specific F test and degrees of freedom. The denominator ratio is adjusted to reflect the full set of error strata: the implied $\mathit{SS}_{\text{error}}$ for the focal effect is its own stratum's residual SS, while the $r_{\text{obs}}$ term includes both measured-factor SS and the residual SS of all other strata. This generalizes the partial-$\eta^2$ CI machinery to the Bakeman (2005) denominator.

References

Algina, J., Keselman, H. J., & Penfield, R. D. (2005). An alternative to Cohen's standardized mean difference effect size: A robust parameter and confidence interval in the two independent groups case. Psychological Methods, 10(3), 317--328. tools:::Rd_expr_doi("10.1037/1082-989X.10.3.317")

Bakeman, R. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37(3), 379--384. tools:::Rd_expr_doi("10.3758/BF03192707")

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York, NY: Chapman & Hall/CRC.

Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. Journal of Statistical Software, 20(8), 1--24. tools:::Rd_expr_doi("10.18637/jss.v020.i08")

Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17, 137--152. tools:::Rd_expr_doi("10.1037/a0028086")

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2027). Designing experiments and analyzing data: A model comparison perspective (4th ed.). Routledge. (See Chapter 3 on $\eta^2$, Chapter 7 on factorial designs, and Chapter 11 on generalized $\eta^2$ for within-subjects designs.)

Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434--447. tools:::Rd_expr_doi("10.1037/1082-989X.8.4.434")

Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605--632. tools:::Rd_expr_doi("10.1177/00131640121971392")

Steiger, J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9(2), 164--182. tools:::Rd_expr_doi("10.1037/1082-989X.9.2.164")

Examples

Run this code

# The pygmalion expectancy experiment: treatment is manipulated, while
# grade is a measured classification, so grade belongs in the denominator.
pyg <- pygmalion
pyg$grade <- factor(pyg$grade)
fit <- aov(iq_8 ~ treatment * grade, data = pyg)

# The default returns the point estimate and leaves the limits NA, because
# both interval methods are approximations.
ci_eta_squared_generalized(fit, observed = "grade")

# The parametric approximation maps a noncentral F interval for the focal
# effect through the observed sums of squares. Warnings mark the method as
# preliminary and report that the interaction's lower limit is clamped at
# 0; treat the limits accordingly.
ci_eta_squared_generalized(fit, observed = "grade", method = "parametric")

# The third option is a residual bootstrap, which refits the model once per
# replication. B = 1000 is the smallest count the function accepts and is
# what keeps this page quick; a reported interval deserves B = 10000 or
# more. The seed makes the interval reproducible and leaves the random
# number stream of the surrounding session as it was.
ci_eta_squared_generalized(fit, observed = "grade",
                           method = "bootstrap", B = 1000, seed = 113)

# Within-subjects ANOVA. The parametric CI uses each effect's own stratum.
set.seed(113)
n <- 20
rm_data <- data.frame(
  subject = factor(rep(seq_len(n), each = 3)),
  time    = factor(rep(c("Pre", "Mid", "Post"), n),
                   levels = c("Pre", "Mid", "Post")),
  y       = rnorm(n, sd = 1.5)[rep(seq_len(n), each = 3)] +
            0.7 * rep(1:3, n) + rnorm(n * 3, sd = 1.2)
)
fit_rm <- aov(y ~ time + Error(subject/time), data = rm_data)
ci_eta_squared_generalized(fit_rm, method = "parametric")

Run the code above in your browser using DataLab