For fixed-effects TSSEM and WLS summaries with positive degrees of freedom,
the RMSEA estimate is
$$\sqrt{G\max(\chi^2-df,0)/(N\,df)}$$
where \(\chi^2\) is the target model statistic. For fixed-effects TSSEM,
\(N\) is the sum of the study sample sizes and \(G\) is the number of
studies. For WLS, \(N\) is the supplied n and \(G=1\);
tssem2() supplies the total sample size from the first stage.
Confidence limits transform the corresponding noncentrality limits using
the same \(N\) and \(G\). The denominator is \(N\), not \(N-G\)
or \(N-1\); this scaling matches the normal-ML convention used by lavaan.
For WLS, the scaling of the supplied sampling covariance matrix should also
be considered when comparing fit statistics. When \(df=0\), RMSEA and
its confidence limits are reported as zero; when \(df<0\), they are NA.
It summaries results for various class.
# S3 method for tssem1FEM
summary(object, ...)
# S3 method for tssem1FEM.cluster
summary(object, ...)
# S3 method for tssem1REM
summary(object, robust = FALSE, ...)
# S3 method for wls
summary(object, df.adjustment = 0, ...)
# S3 method for wls.cluster
summary(object, df.adjustment = 0, ...)
# S3 method for meta
summary(object, homoStat = TRUE, robust = FALSE, ...)
# S3 method for meta3LFIML
summary(object, allX = FALSE, robust = FALSE, ...)
# S3 method for reml
summary(object, ...)
# S3 method for CorPop
summary(object, ...)
# S3 method for Cor3L
summary(object, ...)
# S3 method for bootuniR2
summary(object, probs = c(0, 0.1, 0.5, 0.9, 1), cutoff.chisq.pvalue = 0.05,
cutoff.CFI = 0.9, cutoff.SRMR = 0.1, cutoff.RMSEA = 0.05, ...)
# S3 method for osmasem
summary(object, fitIndices = FALSE, numObs, robust = FALSE, ...)
# S3 method for osmasem2
summary(object, fitIndices = FALSE, numObs, robust = FALSE, mxTryHard = FALSE,
...)
# S3 method for mxsem
summary(object, robust = FALSE, smallSample = FALSE, t.df = NULL, ...)
# S3 method for summary.tssem1FEM
print(x, ...)
# S3 method for summary.wls
print(x, ...)
# S3 method for summary.meta
print(x, ...)
# S3 method for summary.meta3LFIML
print(x, ...)
# S3 method for summary.reml
print(x, ...)
# S3 method for summary.CorPop
print(x, ...)
# S3 method for summary.Cor3L
print(x, ...)
# S3 method for summary.bootuniR2
print(x, ...)
# S3 method for summary.mxsem
print(x, ...)Logical. For osmasem2 fit indices, whether to use
mxTryHard when fitting the comparison models.
An object returned from either class tssem1FEM, class
tssem1FEM.cluster, class tssem1REM, class wls, class
wls.cluster, class meta, class meta3LFIML, class
reml, class mxsem or class CorPop.
An object returned from either class summary.tssem1FEM,
class tssem1FEM.cluster, class summary.wls, class
summary.meta, class summary.meta3LFIML, class
summary.reml or class summary.CorPop.
Logical. Whether to conduct a homogeneity test on the effect sizes.
Logical. Whether to report the predictors and the auxiliary variables.
Logical. Whether to use robust (sandwich) standard errors.
For meta, meta3LFIML, and tssem1REM objects they are
from imxRobustSE. For mxsem objects, see the
section ‘Robust and small-sample inference for mxsem objects’.
Logical, for mxsem objects only. If TRUE
(with robust=TRUE), the cluster-robust covariance of a
single-level clustered fit is also multiplied by \((N-1)/(N-p)\), where
\(N\) is the number of rows and \(p\) the number of free parameters,
including those determined by constraints or at a bound.
Not available for two-level fits. See clusterRobustSE.
NULL or a single positive number, for mxsem
objects only. If NULL (the default), Wald tests and 95% Wald
confidence intervals use the standard normal distribution (\(z\)). If a
number, they use a \(t\) distribution with t.df degrees of
freedom: the \(p\) values and intervals of all parameters and the
delta-method intervals of defined (:=) parameters. The standard
errors themselves are unchanged. Requires intervals.type="z". It
can be combined with model-based or robust standard errors, but it is
intended for cluster-robust inference with few independent clusters; see
the section ‘Robust and small-sample inference for mxsem objects’
for how to choose it.
Numeric. Adjust the degrees of freedom manually. It may
be necessary if the df calculated is incorrect when
diag.constraints=TRUE. For wls.cluster objects, the
adjustment is applied to each fitted model.
Quantiles for the parameter estimates.
Cutoff of the p-value for the chi-square statistic.
The cutoff of the CFI.
The cutoff of the SRMR.
The cutoff of the RMSEA.
Whether to calculate the chi-square statistic and various goodness-of-fit indices in osmasem. Note. It may take a while since statistics of the saturated and independence models are required.
The number of observations in calculating the fit statistics in osmasem. If it is missing, the total number of observations is used.
Further arguments to be passed to
printCoefmat
summary(fit, robust=TRUE) replaces the model-based standard errors
of an mxsem fit by sandwich standard errors: an observation-level
sandwich for a single-level fit without cluster, a cluster
sandwich for a single-level fit with cluster (see
clusterRobustSE), and a cluster-score sandwich for a
two-level fit (see sem). A cluster sandwich is justified as
the number of independent clusters grows (studies in a
meta-analysis, not effect sizes). With few clusters its standard errors
tend to be too small, and \(z\) tests and intervals are then liberal.
t.df replaces the \(z\) reference distribution by a \(t\)
distribution with the given degrees of freedom, which widens the
intervals and raises the \(p\) values when t.df is small. It
does not change the estimates or the standard errors, and it is not
computed automatically, because the appropriate value depends on the
design and the parameter:
Fixed effects of a meta-analysis (the intercept and the
regression coefficients of moderators) with cluster-robust standard
errors: t.df = K - p, where \(K\) is the number of independent
studies (clusters; reported by summary() as “Number of
clusters”, not the number of effect sizes) and \(p\) the number of fixed-effect coefficients
in the mean model, including the intercept. For example, an intercept
and two moderators give \(p=3\). This is the reference distribution
of robust variance estimation proposed by Hedges, Tipton, and Johnson
(2010), and of metafor::robust(). Note that \(p\) counts the
coefficients of the mean model only, not all free parameters as in
smallSample.
Other clustered models: t.df = G - 1, where \(G\)
is the number of independent clusters, is a common convention for
cluster-robust tests (Cameron & Miller, 2015; McNeish, 2026). In
multiple-group models, \(G\) is the total number of clusters.
Limitations:
The same t.df is applied to every parameter. The literature
that justifies it concerns regression coefficients and intercepts of
the mean model. It does not justify tests of variance components,
factor loadings, or other covariance parameters, and it does not
address estimates at or near a bound, whose robust standard errors are
NA.
\(K - p\) can still be too large, and the test liberal, when a moderator is unbalanced or varies mainly between studies (Tipton, 2015). Tests with fewer than about 4 degrees of freedom are unreliable. In those cases, the CR2 correction with Satterthwaite degrees of freedom (Pustejovsky & Tipton, 2018), available in the clubSandwich package for meta-regression models, is preferable.
It does not affect likelihood-ratio tests or model-fit statistics.
References:
Cameron, A. C., & Miller, D. L. (2015). A practitioner's guide to cluster-robust inference. Journal of Human Resources, 50(2), 317--372.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39--65. tools:::Rd_expr_doi("10.1002/jrsm.5")
McNeish, D. (2026). A practical guide to selecting and blending approaches for clustered data: Clustered errors, multilevel models, and fixed-effect models. Psychological Methods, 31(2), 225--251. tools:::Rd_expr_doi("10.1037/met0000620")
Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36(4), 672--683. tools:::Rd_expr_doi("10.1080/07350015.2016.1247004")
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375--393. tools:::Rd_expr_doi("10.1037/met0000011")
Mike W.-L. Cheung <[email protected]>
tssem1, wls,
meta, reml,
rCor, bootuniR2,
osmasem
# \donttest{
## Flat meta-regression of 66 effect sizes nested in 21 clusters, with
## study-clustered robust standard errors.
data(Bornmann07)
Bornmann07$Fellowship <- ifelse(Bornmann07$Type == "Fellowship", 1, 0)
model <- "f =~ 1*logOR
f ~~ tau2*f
logOR ~~ data.v*logOR
f ~ b0*1
logOR ~ b1*Fellowship"
RAM <- lavaan2RAM(model, obs.variables=c("logOR", "Fellowship"),
std.lv=FALSE)
fit <- sem(RAM=RAM, data=Bornmann07, cluster="Cluster")
## z tests and intervals
summary(fit, robust=TRUE)
## t tests and intervals with K - p degrees of freedom: K = 21 clusters
## and p = 2 fixed effects (b0 and b1). Only the rows of b0 and b1 are
## justified by the literature.
K <- length(unique(Bornmann07$Cluster))
summary(fit, robust=TRUE, t.df=K - 2)
# }
Run the code above in your browser using DataLab