permutest
Permutation Tests for 'rma.uni' Objects
The function carries out permutation tests for objects of class "rma.uni"
.
 Keywords
 models
Usage
permutest(x, ...)
"permutest"(x, exact=FALSE, iter=1000, permci=FALSE, progbar=TRUE, retpermdist=FALSE, digits, tol, control, ...)
Arguments
 x
 an object of class
"rma.uni"
.  exact
 logical indicating whether an exact permutation test should be carried out or not (the default is
FALSE
). See ‘Details’.  iter
 integer specifying the number of iterations for the permutation test when not doing an exact test (the default is
1000
iterations).  permci
 logical indicating whether permutationbased CIs should also be calculated (the default is
FALSE
). Can also be a vector of indices specifying for which coefficients a permutationbased CI should be obtained.  progbar
 logical indicating whether a progress bar should be shown (the default is
TRUE
).  retpermdist
 logical indicating whether the permutation distributions of the test statistics should be returned (the default is
FALSE
).  digits
 integer specifying the number of decimal places to which the printed results should be rounded (if unspecified, the default is to take the value from the object).
 tol
 numerical tolerance used when making
>=
or<=< code=""> comparisons between an observed test statistic and the test statistic under the permuted data. Defaults to
.Machine$double.eps^0.5
if unspecified. See ‘Note’.  control
 list of control values for
uniroot
(i.e.,tol
andmaxiter
). Only relevant whenpermci=TRUE
. See ‘Note’.  ...
 other arguments.
Details
For models without moderators, the permutation test is carried out by permuting the signs of the observed effect sizes or outcomes. The (twosided) pvalue of the permutation test is then equal to the proportion of times that the absolute value of the test statistic under the permuted data is as extreme or more extreme than under the actually observed data. See Follmann and Proschan (1999) for more details.
For models with moderators, the permutation test is carried out by permuting the rows of the model matrix (i.e., $X$). The (twosided) pvalue for a particular model coefficient is then equal to the proportion of times that the absolute value of the test statistic for the coefficient under the permuted data is as extreme or more extreme than under the actually observed data. Similarly, for the omnibus test, the pvalue is the proportion of times that the test statistic for the omnibus test is as extreme or more extreme than the actually observed one. See Higgins and Thompson (2004) and Viechtbauer et al. (2015) for more details.
If exact=TRUE
, the function will try to carry out an exact permutation test. An exact permutation test requires fitting the model to each possible permutation once. However, the number of possible permutations increases rapidly with the number of outcomes/studies (i.e., $k$). For models without moderators, there are $2^k$ possible permutations of the signs. Therefore, for $k=5$, there are 32 possible permutations, for $k=10$, there are already 1024, and for $k=20$, there are over one million permutations of the signs.
For models with moderators, the increase in the number of possible permutations may be even more severe. The total number of possible permutations of the model matrix is $k!$. Therefore, for $k=5$, there are 120 possible permutations, for $k=10$, there are 3,628,800, and for $k=20$, there are over $10^18$ permutations of the model matrix.
Therefore, going through all possible permutations may become infeasible. Instead of using an exact permutation test, one can set exact=FALSE
(which is also the default). In that case, the function approximates the exact permutationbased pvalue(s) by going through a smaller number (as specified by the iter
argument) of random permutations. Therefore, running the function twice on the same data can yield (slightly) different pvalues. Setting iter
sufficiently large ensures that the results become stable. Note that if exact=FALSE
and iter
is actually larger than the number of iterations required for an exact permutation test, then an exact test will be carried out.
For models with moderators, the exact permutation test actually only requires fitting the model to each unique permutation of the model matrix. The number of unique permutations will be smaller than $k!$ when the model matrix contains recurring rows. This may be the case when only including categorical moderators (i.e., factors) in the model or when any quantitative moderators included in the model only take on a small number of unique values. When exact=TRUE
, the function therefore uses an algorithm to restrict the test to only the unique permutations of the model matrix, which may make the use of the exact test feasible even when $k$ is large.
When using random permutations, the function ensures that the very first permutation will always correspond to the original data. This avoids pvalues equal to 0.
When permci=TRUE
, the function also tries to obtain permutationbased CIs of the model coefficient(s). This is done by shifting the observed effect sizes or outcomes and finding the most extreme values for which the permutationbased test would just lead to nonrejection. This is computationally very demanding and may take a long time to complete. For models with moderators, one can also set permci
to a vector of indices specifying for which coefficient(s) a permutationbased CI should be obtained. When the algorithm fails to determine a particular CI bound, it will be shown as NA
in the output.
Value

An object of class
"permutest.rma.uni"
. The object is a list containing the following components:The results are formated and printed with the print.permutest.rma.uni
function. One can also use coef.permutest.rma.uni
to obtain the table with the model coefficients, corresponding standard errors, test statistics, pvalues, and confidence interval bounds.
Note
It is important to note that the pvalues obtained with permutation tests cannot reach conventional levels of statistical significance (i.e., $p \le .05$) when $k$ is very small. In particular, for models without moderators, the smallest possible (twosided) pvalue is .0625 when $k=5$ and .03125 when $k=6$. Therefore, the permutation test is only able to reject the null hypothesis at $\alpha=.05$ when $k$ is at least equal to 6. For models with moderators, the smallest possible (twosided) pvalue for a particular model coefficient is .0833 when $k=4$ and .0167 when $k=5$ (assuming that each row in the model matrix is unique). Therefore, the permutation test is only able to reject the null hypothesis at $\alpha=.05$ when $k$ is at least equal to 5. Consequently, permutationbased CIs can also only be obtained when $k$ is sufficiently large.
When the number of permutations required for the exact test is so large as to be essentially indistinguishable from infinity (e.g., factorial(200)
), the function will terminate with an error.
Determining whether a test statistic under the permuted data is as extreme or more extreme than under the actually observed data requires making >=
or <=< code=""> comparisons. To avoid problems due to the finite precision with which computers generally represent numbers, the function uses a numerical tolerance (argument
tol
) when making such comparisons (e.g., instead of sqrt(3)^2  3 >= 0
, which may evaluate to FALSE
, we can use sqrt(3)^2  3 >= 0  .Machine$double.eps^0.5
, which should evaluate to TRUE
).
When obtaining permutationbased CIs, the function makes use of uniroot
. By default, the desired accuracy is set equal to .Machine$double.eps^0.25
and the maximum number of iterations to 100
. The desired accuracy and the maximum number of iterations can be adjusted with the control
argument (i.e., control=list(tol=value, maxiter=value)
).
References
Follmann, D. A., & Proschan, M. A. (1999). Valid inference in random effects metaanalysis. Biometrics, 55, 732737.
Good, P. I. (2009). Permutation, parametric, and bootstrap tests of hypotheses (3rd ed.). New York: Springer.
Higgins, J. P. T., & Thompson, S. G. (2004). Controlling the risk of spurious findings from metaregression. Statistics in Medicine, 23, 16631682.
Viechtbauer, W. (2010). Conducting metaanalyses in R with the metafor package. Journal of Statistical Software, 36(3), 148. http://www.jstatsoft.org/v36/i03/.
Viechtbauer, W., LópezLópez, J. A., SánchezMeca, J., & MarínMartínez, F. (2015). A comparison of procedures to test for moderators in mixedeffects metaregression models. Psychological Methods, 20, 360374.
See Also
Examples
### calculate log relative risks and corresponding sampling variances
dat < escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
### randomeffects model
res < rma(yi, vi, data=dat)
res
### permutation test (approximate and exact)
## Not run:
# permutest(res)
# permutest(res, exact=TRUE)## End(Not run)
### mixedeffects model with two moderators (absolute latitude and publication year)
res < rma(yi, vi, mods = ~ ablat + year, data=dat)
res
### permutation test (approximate only; exact not feasible)
## Not run:
#
# permres < permutest(res, iter=10000, retpermdist=TRUE)
# permres
#
# ### histogram of permutation distribution for absolute latitude
# ### dashed horizontal line: the observed value of the test statistic
# ### red curve: standard normal density
# ### blue curve: kernel density estimate of the permutation distribution
# ### note that the tail area under the permutation distribution is larger
# ### than under a standard normal density (hence, the larger pvalue)
# hist(permres$zval.perm[,2], breaks=120, freq=FALSE, xlim=c(5,5), ylim=c(0,.4),
# main="Permutation Distribution", xlab="Value of Test Statistic", col="gray90")
# abline(v=res$zval[2], lwd=2, lty="dashed")
# abline(v=0, lwd=2)
# curve(dnorm, from=5, to=5, add=TRUE, lwd=2, col=rgb(1,0,0,alpha=.7))
# lines(density(permres$zval.perm[,2]), lwd=2, col=rgb(0,0,1,alpha=.7))
# ## End(Not run)