Power is calculated as the proportion of hypothesis tests which result in a
p-value less than or equal to alpha. e.g.
sum(p <= alpha, na.rm = TRUE) / nsims
Power is defined as the expected probability of rejecting the null hypothesis
for a chosen value of the unknown effect. In a multiple comparisons scenario,
power is defined as the marginal power, which is the expected power of the
test for each individual null hypothesis assumed to be false.
Other forms of power under the multiple comparisons scenario include
disjunctive or conjunctive power. Disjunctive power is defined as the
expected probability of correctly rejecting one or more null hypotheses.
Conjunctive power is defined as the expected probability of correctly
rejecting all null hypotheses. In the simplest case, and where all hypotheses
are independent, if the marginal power is defined as \(\pi\) and \(m\) is
the number of null hypotheses assumed to be false, then disjunctive power may
be calculated as \(1 - (1 - \pi)^m\) and conjunctive power may be calculated
as \(\pi^m\). Disjunctive power tends to decrease with increasingly
correlated hypotheses and conjunctive power tends to increase with
increasingly correlated hypotheses.
Argument ...
... are the name-value pairs for the functions used to perform the tests.
If not named, the functions coerced to character will be used for the
name-value pairs. Typical in non-standard evaluation, ... accepts bare
functions and converts them to a list of expressions. Each element in this
list will be validated as a call and then evaluated on the simulated data.
A base::call() is simply an unevaluated function. Below are some examples
of specifying ... in power().
# Examples of specifying ... in power()
data <- sim_nb(
n1 = 10,
mean1 = 10,
ratio = c(1.6, 2),
dispersion1 = 2,
dispersion2 = 2,
nsims = 200
)# ... is empty, so a an appropriate default function will be provided
power(data)
# This is equivalent to leaving ... empty
power(data, "NB Wald test" = wald_test_nb())
# If not named, "wald_test_nb()" will be used to label the function
power(data, wald_test_nb())
# You can specify any parameters in the call. The data argument
# will automatically be inserted or overwritten.
data |>
power("NB Wald test" = wald_test_nb(equal_dispersion=TRUE, link="log"))
# Multiple functions may be used.
data |>
power(
wald_test_nb(link='log'),
wald_test_nb(link='sqrt'),
wald_test_nb(link='squared'),
wald_test_nb(link='identity')
)
# Just like functions in a pipe, the parentheses are required.
# This will error because wald_test_nb is missing parentheses.
try(power(data, wald_test_nb))
In most cases*, any user created test function may be utilized in ... if the
following conditions are satisfied:
The function contains argument data which is defined as a list with the
first and second elements for simulated data.
The return object is a list with element p for the p-value of the
hypothesis test.
Validate with test cases beforehand.
*Simulated data of class log_lognormal_mixed_two_sample has both independent
and dependent data. To ensure the appropriate test function is used,
power.log_lognormal_mixed_two_sample() allows only
t_test_welch() and t_test_paired() in .... Each will
be evaluated on the simulated data according to column data$cor. If one or
both of these functions are not included in ..., the corresponding default
function will be used automatically. If any other test function is included,
an error will be returned.
Argument alpha
\(\alpha\) is known as the type I assertion probability and is defined as
the expected probability of rejecting a null hypothesis when it was actually
true. \(\alpha\) is compared with the p-value and used as the decision
boundary for rejecting or not rejecting the null hypothesis.
The family-wise error rate is the expected probability of making one or more
type I assertions among a family of hypotheses. Using Bonferroni's method,
\(\alpha\) is chosen for the family of hypotheses then divided by the
number of tests performed (\(m\)). Each individual hypothesis is tested at
\(\frac{\alpha}{m}\). For example, if you plan to conduct 30 hypothesis
tests and want to control the family-wise error rate to no greater than
\(\alpha = 0.05\), you would set alpha = 0.05/30.
The per-family error rate is the expected number of type I assertions among a
family of hypotheses. If you calculate power for the scenario where you
perform 1,000 hypotheses and want to control the per-family error rate to no
greater than 10 type I assertions, you would choose alpha = 10/1000. This
implicitly assumes all 1,000 hypotheses are truly null. Alternatively, if you
assume 800 of these hypotheses are truly null and 200 are not,
alpha = 10/1000 would control the per-family error rate to no greater than
8 type I assertions. If it is acceptable to keep the per-family error rate as
10, setting alpha = 10/800 would provide greater marginal power than the
previous scenario.
These two methods assume that the distribution of p-values for the truly null
hypotheses are uniform(0,1), but remain valid under various other testing
scenarios (such as dependent tests). Other multiple comparison methods, such
as FDR control, are common in practice but don't directly fit into this
power simulation framework.
Column nsims
The final number of valid simulations per unique set of simulation parameters
may be less than the original number requested. This may occur when the test
results in a missing p-value. For wald_test_bnb(), pathological MLE
estimates, generally from small sample sizes and very small dispersions, may
result in a negative estimated standard deviation of the ratio. Thus the test
statistic and p-value would not be calculated. Note that simulated data from
sim_nb() and sim_bnb() may also reduce nsims during the data simulation
phase.
The nsims column in the return data frame is the effective number of
simulations for power results.