`sca_test()` performs the permutation-based joint-inference test of Simonsohn, Simmons, and Nelson (2020). It tests the sharp null hypothesis that the focal independent variable `x` has no effect on `y` in any specification. The focal variable is repeatedly permuted (shuffled) and the entire specification curve is re-estimated on each permuted data set, building a null distribution for curve-level summary statistics. The observed statistics are then compared to that null distribution to obtain permutation p-values.
The permutation is *blocked within the first fixed effect* when `fixed_effects` are supplied, because exchangeability of `x` only holds within fixed-effect groups; otherwise it is a global permutation.
sca_test(
y,
x,
controls,
data,
weights = NULL,
family = "linear",
link = NULL,
fixed_effects = NULL,
n_permutations = 500,
test_stats = c("median", "share_significant", "stouffer"),
direction = "two.sided",
alpha = 0.05,
sca_data = NULL,
keep_curves = FALSE,
null_type = c("shuffle_x", "freedman_lane", "residual_bootstrap"),
common_sample = FALSE,
parallel = FALSE,
workers = 2,
seed = NULL,
progress_bar = TRUE
)An object of class `"sca_test"`: a list with elements `observed` (a named vector of observed statistics), `null_distribution` (a data frame with one row per usable permutation and one column per statistic, plus `n_valid`), `p_values` (a named vector of permutation p-values), `params` (a list of run metadata), and, when `keep_curves = TRUE`, `null_curves` (a list with `spec`, a data frame of each specification's key and observed coefficient, and `null_coef`, a specifications-by-permutations matrix of the permuted focal coefficients). When `keep_curves = TRUE` *and* the null is confound-preserving (`null_type = "freedman_lane"` or `"residual_bootstrap"`), `null_curves` also gains `fwer`, the per-specification family-wise-error-rate-adjusted p-values attached automatically by [sca_minp()] (see there for its structure).
A string with the dependent variable column name, or a two-sided formula specifying the whole model (`y ~ x + control1 + control2 | fixed_effect`), exactly as accepted by [sca()]. When a formula is supplied `data` may be passed positionally.
A string with the focal independent variable column name.
A vector of strings with the control variable column names.
A data frame containing the variables.
Optional string naming a weights column in `data`. Weights are held fixed to their row and never permuted; this assumes the weights are exogenous to `x`.
A string giving the model family, as in [sca()]. Defaults to `"linear"`.
A string giving the link function, as in [sca()]. Defaults to `NULL`.
A string (or vector) naming fixed-effect variables, as in [sca()]. When supplied, `x` is permuted within levels of the first fixed effect.
An integer number of permutations used to build the null distribution. Defaults to `500`. The smallest attainable p-value is `1 / (n_permutations + 1)`.
A character vector of statistics to compute. Any of `"median"` (median focal coefficient across specifications), `"share_significant"` (share of specifications statistically significant in the reference direction), `"stouffer"` (Stouffer's combined Z over the per-specification p-values), and the descriptive `"share_sign"` (share of specifications with the reference sign). Defaults to `c("median", "share_significant", "stouffer")`.
One of `"two.sided"` (default), `"positive"`, or `"negative"`, giving the a-priori predicted direction of the effect. The package never chooses the direction from the data: with `"two.sided"` the tests are direction-agnostic; use `"positive"`/`"negative"` only for a genuinely a-priori prediction.
The significance threshold used by `"share_significant"`. Defaults to `0.05`.
Optional data frame previously returned by [sca()] for the *observed* model, used purely to skip recomputing the observed curve. The null distribution is always recomputed from `data`, so `data` remains required. Its control indicator columns must be consistent with `controls`. Ignored (with a warning) for `null_type = "freedman_lane"` or `"residual_bootstrap"`, where the observed curve is recomputed on the forced common sample so it matches the null.
A boolean indicating whether to retain, for every specification, the focal coefficient from each permuted curve (aligned across permutations by specification, not row order). Required by [plot_sca_test_specs()]; increases the size of the returned object. Defaults to `FALSE`.
How the null is generated. One of:
(default) Simonsohn-Simmons-Nelson sharp-null permutation: shuffle the focal variable (blocked within the first fixed effect). Appropriate for experimental / as-if-randomly-assigned `x`. It breaks `x`'s correlation with the controls, so it is miscalibrated (anti-conservative) under collinearity -- the observational case.
Freedman-Lane (1983) partial permutation. A reduced model `y ~ controls` (the full control superset, `x` omitted, fixed effects retained) is fit once; its residuals are permuted (blocked within the first fixed effect) and added back to its fitted values to form a null response, on which the whole curve is refit. This preserves `x`'s partial correlation with the controls. It tests the sharp null of no partial effect of `x` given the superset controls, so under-controlled specifications may have non-zero null centres by design.
The SSN (2020) observational scheme: impose the null on the response (`y - betahat * x`, with `betahat` from the full control-superset specification), then resample rows with replacement and refit. A null-imposed case bootstrap (nearly equivalent to Flachaire 1999); robust to heteroskedasticity but its p-values carry extra Monte-Carlo variability.
The `"freedman_lane"` and `"residual_bootstrap"` nulls are defined for `family = "linear"` only (including fixed effects) and force `common_sample = TRUE`.
A boolean passed through to [sca()]: fit every specification on the rows complete across all model variables. Defaults to `FALSE`; forced to `TRUE` for the `"freedman_lane"` and `"residual_bootstrap"` nulls.
A boolean indicating whether to parallelise the permutations. Defaults to `FALSE`. The inner [sca()] call is always run serially to avoid nested parallelism.
An integer number of parallel workers. Defaults to `2`.
Optional integer seed. When supplied, results are reproducible and identical for the serial and parallel paths (the permutations are generated up front).
A boolean indicating whether to show a progress bar for the permutations. Defaults to `TRUE`.
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4, 1208-1214. tools:::Rd_expr_doi("10.1038/s41562-020-0912-z")
Freedman, D., & Lane, D. (1983). A nonstochastic interpretation of reported significance levels. Journal of Business & Economic Statistics, 1(4), 292-298.
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014). Permutation inference for the general linear model. NeuroImage, 92, 381-397.
[plot_sca_test()] to visualise the null distributions of the test statistics, and [plot_sca_test_specs()] for the per-specification null-band plot (requires `keep_curves = TRUE`).
# \donttest{
# Test whether temperature robustly predicts salinity across specifications.
result <- sca_test(y = "Salnty", x = "T_degC",
controls = c("ChlorA", "O2Sat"),
data = bottles, n_permutations = 100, progress_bar = FALSE)
result
# }
Run the code above in your browser using DataLab