ss_aipe_reliability: Sample Size Planning for Accuracy in Parameter Estimation for Reliability Coefficients

Description

Computes the necessary sample size for the confidence interval on a population reliability coefficient (coefficient alpha or coefficient omega, depending on the assumed measurement model) to have expected width no larger than width, or (when assurance is supplied) to be no wider than width with the specified probability. This is the accuracy in parameter estimation (AIPE) counterpart to power based planning for reliability and is the companion of reliability. The closed form planner uses the Terry and Kelley (2012) formulas under the user-selected measurement model and confidence interval type; when assurance is supplied, the function follows the closed form with an internal Monte Carlo simulation to find the smallest N delivering the requested assurance.

Coefficient alpha (Guttman, 1945; subsequently popularized by Cronbach, 1951) is returned for the parallel and tau equivalent (i.e., the so called True Score) measurement models; coefficient omega (McDonald, 1999) is returned for the congeneric model.

Usage

ss_aipe_reliability(
  model = NULL,
  type = NULL,
  width = NULL,
  S = NULL,
  conf_level = 0.95,
  assurance = NULL,
  data = NULL,
  i = NULL,
  cor_est = NULL,
  lambda = NULL,
  psi_square = NULL,
  initial_iter = 500,
  final_iter = 5000,
  start_ss = NULL,
  verbose = FALSE
)

Value

A data.frame with columns term and value. Without assurance the data frame has a single row, "necessary_N", giving the necessary N. With assurance

supplied, the data frame has five rows: "necessary_N" (necessary N), "width" (echo of the target width), "specified_assurance" (echo of the requested probability), "empirical_assurance" (the assurance achieved at the returned N in the Monte Carlo verification), and "final_iter"

(number of Monte Carlo iterations used).

Arguments

model: The measurement model assumed for the population. Accepts (case-sensitive aliases shown in parentheses): "Parallel" ("parallel", "SB", "Spearman Brown", "Spearman-Brown", "sb") for the strictly parallel items model; "True Score" ("True Score Equivalent", "True-Score Equivalent", "Equivalent", "Tau Equivalent", "tau-equivalent", "Tau-Equivalent", "True-Score", "true-score", "true score", "Cronbach", "cronbach", "Chronbach", "alpha") for the tau equivalent model (in which case the function plans for coefficient alpha); or "Congeneric" ("congeneric", "omega", "Omega") for the congeneric model (in which case the function plans for coefficient omega).
type: The method used to construct the confidence interval on the reliability coefficient: either "Factor Analytic" (McDonald, 1999), available for all three measurement models, or "Normal Theory" (van Zyl, Neudecker, & Nel, 2000), available for the parallel and tau equivalent models only.
width: The desired full width of the two-sided confidence interval.
S: A symmetric population covariance (or correlation) matrix among the items, used to imply the population reliability and its sampling distribution.
conf_level: Confidence level (i.e., \(1 - \alpha\), where \(\alpha\) is the Type I error rate). Default 0.95.
assurance: Optional probability with which the realized interval is to be no wider than width. When NULL (the default), the planner targets the expected width; when supplied (e.g., 0.80, 0.85, 0.95), the function follows the closed form with a Monte Carlo search.
data: A data set from which the population covariance matrix should be inferred.
i: Number of items.
cor_est: The presumed inter-item correlation. One value for the parallel and tau equivalent models.
lambda: Vector of population factor loadings.
psi_square: Vector of population unique (error) variances.
initial_iter: Number of Monte Carlo iterations used in the initial assurance search.
final_iter: Number of Monte Carlo iterations used in the final assurance verification.
start_ss: Optional starting sample size for the iterative assurance search.
verbose: If TRUE, prints the current sample size and empirical assurance at each step of the Monte Carlo search.

Warning

In some conditions the factor analytic model fit by cfa_1 (via lavaan) may fail to converge, and you may see a non-convergence message from lavaan. The Monte Carlo assurance search treats a non-converged iteration as missing and continues, so a few such messages do not invalidate the result. Frequent non-convergence usually means the model is poorly determined by the data, for example because of a small sample size, a low number of iterations, or a poorly behaved covariance matrix.

Author

Ken Kelley [email protected]

Details

The Monte Carlo assurance search simulates covariance matrices from the population implied by the inputs and, at each candidate sample size, computes the realized confidence interval width with the same machinery the estimation side of the package uses. For type = "Factor Analytic" the single-factor model is fit by maximum likelihood (equal loadings for the parallel and tau equivalent models, free loadings for the congeneric model) and the interval is the delta method Wald interval on the model implied reliability, the interval of reliability_omega(denominator = "model_implied", ci_method = "ml") and of reliability_alpha(estimator = "model_implied", ci_method = "ml"). For type = "Normal Theory" the interval uses the van Zyl, Neudecker, and Nel (2000) closed form standard error for the tau equivalent model and its compound symmetry simplification for the parallel model, the same closed form behind reliability_alpha(ci_method = "ml"). The congeneric model has no normal theory form, so type = "Normal Theory" is an error there.

The cost of the assurance search depends on the interval. The normal theory intervals are closed forms, so a search at the default initial_iter = 500 and final_iter = 5000 finishes in seconds. With the factor analytic interval a one factor model is fit at every Monte Carlo iteration and at every candidate sample size the search visits, so a congeneric plan with an assurance runs for tens of seconds even at initial_iter = 50 and final_iter = 200 and for many minutes at the defaults. The examples on this page therefore stop at the closed form for the congeneric plan; the assurance version is the same call with assurance supplied (for example, assurance = .80), and set.seed() before the call makes the search reproducible.

References

Kelley, K., & Cheng, Y. (2012). Estimation of and confidence interval formation for reliability coefficients of homogeneous measurement instruments. Methodology, 8, 39--50. tools:::Rd_expr_doi("10.1027/1614-2241/a000036")

Kelley, K., & Pornprasertmanit, S. (2016). Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21, 69--92. tools:::Rd_expr_doi("10.1037/a0040086")

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2027). Designing experiments and analyzing data: A model comparison perspective (4th ed.). Routledge.

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum Associates.

Terry, L. J., & Kelley, K. (2012). Sample size planning for composite reliability coefficients: Accuracy in parameter estimation via narrow confidence intervals. British Journal of Mathematical and Statistical Psychology, 65, 371--401. tools:::Rd_expr_doi("10.1111/j.2044-8317.2011.02030.x")

van Zyl, J. M., Neudecker, H., & Nel, D. G. (2000). On the distribution of the maximum likelihood estimator of Cronbach's alpha. Psychometrika, 65(3), 271--280. tools:::Rd_expr_doi("10.1007/BF02296146")

Examples

Run this code

# Expected confidence interval width (closed form, no Monte Carlo search).
ss_aipe_reliability(model = "Parallel", type = "Normal Theory", width = .1,
  i = 6, cor_est = .3, psi_square = .2, conf_level = .95, assurance = NULL)

# The assurance cases run a Monte Carlo search. initial_iter = 50 and
# final_iter = 200 keep the examples quick; a reported plan deserves the
# defaults of 500 and 5000, and set.seed() makes the search reproducible.
set.seed(113)

# Same population, now targeting an assurance.
ss_aipe_reliability(model = "Parallel", type = "Normal Theory", width = .1,
  i = 6, cor_est = .3, psi_square = .2, conf_level = .95, assurance = .85,
  initial_iter = 50, final_iter = 200)

# The true score (tau equivalent) model takes psi_square as a vector of
# length i (number of items) while cor_est stays a single value.
ss_aipe_reliability(model = "True Score", type = "Normal Theory",
  width = .1, i = 5, cor_est = .3, psi_square = c(.2, .3, .3, .2, .3),
  conf_level = .95, assurance = .85, initial_iter = 50, final_iter = 200)

# Congeneric model, planned from the item loadings and error variances rather
# than from a single correlation. With assurance = NULL the necessary N comes
# from the closed form expected width evaluated at the implied population
# correlation matrix, so type does not enter the answer; type selects the
# interval that the Monte Carlo assurance search evaluates. Adding an
# assurance to this plan fits a one factor model at every Monte Carlo
# iteration and is the slow case, so the page stops at the closed form
# (see Details).
ss_aipe_reliability(model = "Congeneric", type = "Factor Analytic", width = .15,
  i = 4, lambda = c(.8, .7, .7, .8), psi_square = c(.4, .5, .5, .4),
  conf_level = .95, assurance = NULL)

# Planning from a presumed population correlation matrix among the items.
pop_mat <- rbind(
  c(1.0000000, 0.3813850, 0.4216370, 0.3651484, 0.4472136),
  c(0.3813850, 1.0000000, 0.4020151, 0.3481553, 0.4264014),
  c(0.4216370, 0.4020151, 1.0000000, 0.3849002, 0.4714045),
  c(0.3651484, 0.3481553, 0.3849002, 1.0000000, 0.4082483),
  c(0.4472136, 0.4264014, 0.4714045, 0.4082483, 1.0000000))
ss_aipe_reliability(model = "True Score", type = "Normal Theory", width = .15,
  S = pop_mat, conf_level = .95, assurance = .85, initial_iter = 50,
  final_iter = 200)

Run the code above in your browser using DataLab