mcse: Monte Carlo Standard Errors for Tests

Description

This function calculates Monte Carlo standard errors for (non-exact) nonparametric tests. The MCSEs can be used to determine (i) the accuracy of a test for a given number of resamples, or (ii) the number of resamples needed to achieve a test with a given accuracy.

Usage

mcse(R, delta, conf.level = 0.95, sig.level = 0.05,
     alternative = c("two.sided", "one.sided"))

Value

mcse: Monte Carlo standard error.
R: Number of resamples.
delta: Accuracy of approximation.
conf.level: Confidence level.
sig.level: Significance level.
alternative: Alternative hypothesis.

Arguments

R: Number of resamples (positive integer).
delta: Accuracy of the approximation (number between 0 and 1).
conf.level: Confidence level for the approximation (number between 0 and 1).
sig.level: Significance level of the test (number between 0 and 1).
alternative: Alternative hypothesis (two-sided or one-sided).

Author

Nathaniel E. Helwig <helwig@umn.edu>

Details

Note: either R or delta must be provided.

Let $F(x)$ denote the distribution function for the full permutation distribution, and let $G(x)$ denote the approximation obtained from $R$ resamples. The Monte Carlo standard error is given by $$ \sigma(x) = \sqrt{ F(x) [1 - F(x)] / R } $$ which is the standard deviation of $G(x)$.

A symmetric confidence interval for $F(x)$ can be approximated as $$ G(x) +/- C \sigma(x) $$ where $C$ is some quantile of the standard normal distribution. Note that the critical value $C$ corresponds to the confidence level (conf.level) of the approximation.

Let $\alpha$ denote the significance level (sig.level) for a one-sided test ($\alpha$ is one-half the significance level for two-sided tests). Define $a$ to be the value of the test statistic such that $F(a) = \alpha$.

The parameter $\delta$ (delta) quantifies the accuracy of the approximation, such that $$ |G(a) - \alpha| < \alpha \delta $$ with a given confidence, which is controlled by the conf.level argument.

References

Helwig, N. E. (2019). Statistical nonparametric mapping: Multivariate permutation tests for location, correlation, and regression problems in neuroimaging. WIREs Computational Statistics, 11(2), e1457. doi: 10.1002/wics.1457

Examples

Run this code


###***###   EXAMPLE 1   ###***###

# get the Monte Carlo standard error and the 
# accuracy (i.e., delta) for given R = 10000
# using the default two-sided alternative hypothesis,
# the default confidence level (conf.level = 0.95),
# and the default significance level (sig.level = 0.05)

mcse(R = 10000)

# se = 0.0016
# delta = 0.1224


###***###   EXAMPLE 2   ###***###

# get the Monte Carlo standard error and the 
# number of resamples (i.e., R) for given delta = 0.01
# using a one-sided alternative hypothesis,
# the default confidence level (conf.level = 0.95),
# and the default significance level (sig.level = 0.05)

mcse(delta = 0.1, alternative = "one.sided")

# se = 0.0026
# R = 7299

Run the code above in your browser using DataLab