safestats (version 0.8.7)

replicateTTests: Simulate Early Stopping Experiments

Description

Simulate multiple data sets to show the effects of optional testing for safe (and frequentist) tests.

Usage

replicateTTests(
  nPlan,
  deltaTrue,
  muGlobal = 0,
  sigmaTrue = 1,
  paired = FALSE,
  alternative = c("twoSided", "greater", "less"),
  lowN = 3,
  nSim = 1000L,
  alpha = 0.05,
  beta = 0.2,
  safeOptioStop = TRUE,
  parameter = NULL,
  freqOptioStop = FALSE,
  nPlanFreq = NULL,
  logging = TRUE,
  seed = NULL,
  pb = TRUE,
  ...
)

Value

Returns an object of class "safeTSim". An object of class "safeTSim" is a list containing at least the following components:

nPlan

the planned sample size(s).

deltaTrue

the value of the true standardised effect size (test-relevant parameter) provided by the user.

muGlobal

the true global mean of a paired or two-sample t-test (nuisance parameter) provided by the user.

paired

if TRUE then paired t-test.

alternative

any of "twoSided", "greater", "less" provided by the user.

lowN

the smallest number of samples (first group) at which monitoring of the tests begins.

nSim

the number of replications of the experiment.

alpha

the tolerable type I error provided by the user.

beta

the tolerable type II error provided by the user.

testType

any of "oneSample", "paired", "twoSample" provided by the user.

parameter

the parameter (point prior) used in the safe test derived from the design. Acquired from designSafeT().

nPlanFreq

the frequentist planned sample size(s). Acquired from designFreqT()

safeSim

list with the simulation results of the safe test under optional stopping.

freqSim

list with the simulation results of the frequentist test under optional stopping.

Arguments

nPlan

vector of max length 2 representing the planned sample sizes.

deltaTrue

numeric, the value of the true standardised effect size (test-relevant parameter).

muGlobal

numeric, the true global mean of a paired or two-sample t-test. Its value should not matter for the test. This parameter is treated as a nuisance.

sigmaTrue

numeric > 0,the true standard deviation of the data. Its value should not matter for the test.This parameter treated is treated as a nuisance.

paired

logical, if TRUE then paired t-test.

alternative

a character string specifying the alternative hypothesis must be one of "twoSided" (default), "greater" or "less".

lowN

integer that defines the smallest n of our search space for n.

nSim

the number of replications, that is, experiments with max samples nPlan.

alpha

numeric in (0, 1) that specifies the tolerable type I error control --independent of n-- that the designed test has to adhere to. Note that it also defines the rejection rule e10 > 1/alpha.

beta

numeric in (0, 1) that specifies the tolerable type II error control necessary to calculate both the sample sizes and deltaS, which defines the test. Note that 1-beta defines the power.

safeOptioStop

logical, TRUE implies that optional stopping simulation is performed for the safe test.

parameter

numeric, the safe test defining parameter, i.e., deltaS (use designSafeT to find this).

freqOptioStop

logical, TRUE implies that optional stopping simulation is performed for the frequentist test.

nPlanFreq

the frequentist sample size(s) to plan for. Acquired from designFreqT().

logging

logical, if TRUE, then return the simulated data.

seed

To set the seed for the simulated data.

pb

logical, if TRUE, then show progress bar.

...

further arguments to be passed to or from methods.

Examples

# Design safe test
alpha <- 0.05
beta <- 0.20
designObj <- designSafeT(1, alpha=alpha, beta=beta)

# Design frequentist test
freqObj <- designFreqT(1, alpha=alpha, beta=beta)

# Simulate under the alternative with deltaTrue=deltaMin
simResults <- replicateTTests(nPlan=designObj$nPlan, deltaTrue=1, parameter=designObj$parameter,
                              nPlanFreq=freqObj$nPlan, beta=beta, nSim=250)

# Should be about 1-beta
simResults$safeSim$powerAtN1Plan

# This is higher due to optional stopping
simResults$safeSim$powerOptioStop

# Optional stopping allows us to do better than n1PlanFreq once in a while
simResults$safeSim$probLeqN1PlanFreq
graphics::hist(simResults$safeSim$allN, main="Histogram of stopping times", xlab="n1",
               breaks=seq.int(designObj$nPlan[1]))

# Simulate under the alternative with deltaTrue > deltaMin
simResults <- replicateTTests(nPlan=designObj$nPlan, deltaTrue=1.5, parameter=designObj$parameter,
                              nPlanFreq=freqObj$nPlan, beta=beta, nSim=250)

# Should be larger than 1-beta
simResults$safeSim$powerAtN1Plan

# This is even higher due to optional stopping
simResults$safeSim$powerOptioStop

# Optional stopping allows us to do better than n1PlanFreq once in a while
simResults$safeSim$probLeqN1PlanFreq
graphics::hist(simResults$safeSim$allN, main="Histogram of stopping times", xlab="n1",
               breaks=seq.int(designObj$nPlan[1]))

# Under the null deltaTrue=0
simResults <- replicateTTests(nPlan=designObj$nPlan, deltaTrue=0, parameter=designObj$parameter,
                              nPlanFreq=freqObj$nPlan, freqOptioStop=TRUE, beta=beta, nSim=250)

# Should be lower than alpha, because if the null is true, P(S > 1/alpha) < alpha for all n
simResults$safeSim$powerAtN1Plan

# This is a bit higher due to optional stopping, but if the null is true,
# then still P(S > 1/alpha) < alpha for all n
simResults$safeSim$powerOptioStop

# Should be lowr than alpha, as the experiment is performed as was planned
simResults$freqSim$powerAtN1Plan

# This is larger than alpha, due to optional stopping.
simResults$freqSim$powerOptioStop
simResults$freqSim$powerOptioStop > alpha