Quantifies how much misspecification of the population ICC can distort an
AIPE-based sample size plan. Given a true (population) ICC and the value
used in planning, the function simulates draws of size \(n \times k\)
from the relevant variance-components model, computes the ICC and its
F-distribution confidence interval on each replication via
icc, and summarizes how often the realized interval width
is below the desired target and how often the interval covers the
population value. This is the standard sensitivity-analysis workflow
described in Kelley (2007) and Maxwell, Delaney, and Kelley (2027,
Section 3.11 on sample size planning).
ss_aipe_icc_sensitivity(
true_rho = NULL,
estimated_rho = NULL,
k,
width,
assurance = NULL,
specified_N = NULL,
conf_level = 0.95,
type = c("ICC(1,1)", "ICC(2,1)", "ICC(3,1)", "ICC(1,k)", "ICC(2,k)", "ICC(3,k)"),
G = 1000,
print_iter = FALSE,
filename = NULL
)A data.frame with columns term and value
summarizing the Monte Carlo results across the G replications.
The term entries are: "mean_icc", "median_icc",
"sd_icc" (mean / median / SD of the G observed ICC
estimates); "mean_ci_width", "median_ci_width",
"sd_ci_width" (corresponding summaries of the realized interval
widths); "pct_ci_less_w" (proportion of intervals at or below
the planning width width); "pct_ci_miss_low" and
"pct_ci_miss_high" (tail-specific non-coverage of
true_rho); "total_type_I_error" (overall empirical
non-coverage of true_rho); plus the input echoes
"total_N", "k", "true_rho",
"estimated_rho", "width", "conf_level", and
"assurance" (present only when an assurance was supplied).
The ICC type is not a row; it is stored as the "icc_type"
attribute on the returned object so the value column stays
numeric.
Population intraclass correlation coefficient (the
data generating value), at the level matching type. Must lie in
\([0, 1)\).
ICC used to plan the study (the value the
researcher guessed when invoking ss_aipe_icc). Supply
this or specified_N but not both.
Number of raters (or repeated measurements) per subject; must be at least 2.
Desired full width of the (back-transformed) confidence interval on the ICC.
Probability with which the realized interval should be
no wider than width (must be NULL or strictly between
0 and 1; NULL means plan to the expected width without
an assurance constraint).
Pre-specified number of subjects to evaluate (use this
when you want the sensitivity results at a fixed n rather than
at the n that ss_aipe_icc would recommend);
incompatible with estimated_rho.
Desired confidence level (i.e., 1 minus the Type I error rate); default 0.95.
Which Shrout-Fleiss (1979) ICC form is being planned. One
of "ICC(1,1)" (default; one-way random model),
"ICC(2,1)" (two-way random model, absolute agreement),
"ICC(3,1)" (two-way mixed model, consistency), or the
average-of-\(k\) versions "ICC(1,k)", "ICC(2,k)",
"ICC(3,k)". The simulation generates data from the variance-
components model matching the requested type, so the realized ICC
on each replication is comparable to true_rho.
Number of Monte Carlo replications; defaults to 1000. Increase (e.g., 5000 or 10000) for stable empirical-coverage estimates.
Logical. If TRUE the simulation prints the
iteration index after each replication (helpful for long runs);
default FALSE.
An optional path for a comma separated file recording
every replication (the realized ICC, the two confidence limits, the
interval width, and the two tail-specific non-coverage indicators):
nothing is written when filename is NULL (the default),
a new file with a header row is created otherwise, an existing file
at that path is appended to, and a throwaway run should point it at
tempfile(fileext = ".csv").
Ken Kelley [email protected]
Sample size planning for the intraclass correlation coefficient under
the Accuracy in Parameter Estimation framework chooses n so that
the expected (or, with assurance, the high-probability) confidence
interval width is no larger than width (Bonett, 2002;
Kelley & Maxwell, 2003). Because the procedure assumes the planning
value estimated_rho matches the population value
true_rho, in practice the realized width will deviate from the
planned width whenever the planning value is wrong. This sensitivity
analysis quantifies the deviation by Monte Carlo simulation: the
planned n is obtained from ss_aipe_icc with
estimated_rho, then samples are drawn from the true
population (with population ICC true_rho) and the realized
confidence interval widths are summarized.
Data generating model. For type starting with
"ICC(1," the simulation uses the one-way random model: each row
(subject) gets a subject random effect, every cell adds independent
Gaussian noise, and the population ICC equals
\(\sigma^2_{\mathrm{subj}} / (\sigma^2_{\mathrm{subj}} +
\sigma^2_{\mathrm{err}})\). For type starting with "ICC(2,"
the simulation also adds a rater random effect, so the population
ICC(2,1) equals \(\sigma^2_{\mathrm{subj}} /
(\sigma^2_{\mathrm{subj}} + \sigma^2_{\mathrm{rater}} +
\sigma^2_{\mathrm{err}})\). For type starting with "ICC(3,"
the rater effect is fixed (centered constants), so the population
ICC(3,1) equals \(\sigma^2_{\mathrm{subj}} /
(\sigma^2_{\mathrm{subj}} + \sigma^2_{\mathrm{err}})\) but the
two-way decomposition is used in the estimator. Within each cell, the
total variance is unity by construction. For the single-rater forms
true_rho therefore maps directly to the subject-variance share;
for the average-of-\(k\) forms true_rho is first mapped to the
single-rater scale through the inverse Spearman-Brown relation
\(\rho = \rho_k / [k - (k - 1)\rho_k]\), so that the population ICC
at the average-of-\(k\) level equals true_rho. Coverage is
checked against true_rho on its own scale, matching the scale on
which icc reports each form.
Bonett, D. G. (2002). Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine, 21(9), 1331--1335. tools:::Rd_expr_doi("10.1002/sim.1108")
Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. Journal of Statistical Software, 20(8), 1--24. tools:::Rd_expr_doi("10.18637/jss.v020.i08")
Kelley, K., & Maxwell, S. E. (2003). Sample size for multiple regression: Obtaining regression coefficients that are accurate, not simply significant. Psychological Methods, 8(3), 305--321. tools:::Rd_expr_doi("10.1037/1082-989X.8.3.305")
Maxwell, S. E., Delaney, H. D., & Kelley, K. (2027). Designing experiments and analyzing data: A model comparison perspective (4th ed.). Routledge. (See Chapters 10 and 11 on intraclass correlation and reliability.)
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420--428.
ss_aipe_icc, icc, var_icc
design_consequences for what a chosen design delivers:
power, the Type S (sign) and Type M (exaggeration) errors of the
significance filter, and the expected confidence interval width.
Other AIPE sample size planning:
ss_aipe_c_sensitivity(),
ss_aipe_cliff_delta(),
ss_aipe_cliff_delta_sensitivity(),
ss_aipe_composite_sem(),
ss_aipe_equivalence_r(),
ss_aipe_equivalence_r_sensitivity(),
ss_aipe_equivalence_smd(),
ss_aipe_equivalence_smd_sensitivity(),
ss_aipe_icc(),
ss_aipe_indirect_effect(),
ss_aipe_indirect_effect_sensitivity(),
ss_aipe_mixed_effects_sensitivity(),
ss_aipe_omega_squared(),
ss_aipe_omega_squared_sensitivity(),
ss_aipe_partial_r(),
ss_aipe_partial_r_sensitivity(),
ss_aipe_pcm_sensitivity(),
ss_aipe_r(),
ss_aipe_r_sensitivity(),
ss_aipe_reliability_sensitivity(),
ss_aipe_semipartial_r(),
ss_aipe_semipartial_r_sensitivity()
# Reduced G and a wide target width keep this Monte Carlo example fast;
# raise G (e.g., 1000 or more) for stable empirical-coverage estimates.
# Well-specified case: plan with the true population ICC.
set.seed(113)
ss_aipe_icc_sensitivity(
true_rho = 0.70,
estimated_rho = 0.70,
k = 3,
width = 0.40,
conf_level = 0.95,
G = 25,
print_iter = FALSE
)
# Misspecified case: the planner used .70 but the truth is .50.
# The realized interval widths will tend to be wider than the target.
set.seed(113)
ss_aipe_icc_sensitivity(
true_rho = 0.50,
estimated_rho = 0.70,
k = 3,
width = 0.40,
conf_level = 0.95,
G = 25,
print_iter = FALSE
)
# Fixed-n mode: skip the planner and evaluate at a chosen sample size.
set.seed(113)
ss_aipe_icc_sensitivity(
true_rho = 0.50,
specified_N = 40,
k = 3,
width = 0.40,
G = 25,
print_iter = FALSE
)
Run the code above in your browser using DataLab