combine_p: Combine Independent P-Values Across Studies

Description

Combines the one-tailed p-values from \(k\) independent tests of a common directional hypothesis into a single test, by any of the four classical methods that Raudenbush (1984) applied to the teacher expectancy experiments: Fisher's (1938) “adding logs” chi square, Edgington's (1972) “adding ps”, the Mosteller and Bush (1954) “adding Zs” (Stouffer) method, and a weighted adding-Zs variant (weights are typically the studies' degrees of freedom). Combined significance tests answer the narrow question “is there an effect in at least some studies?”; they do not estimate its size. Pair them with meta_smd or meta_es for estimation, which is almost always the more informative summary.

Usage

combine_p(
  p,
  method = c("fisher", "edgington", "stouffer", "stouffer_weighted"),
  weights = NULL
)

Value

A data.frame (class dmar_tbl) with, per requested method, its statistic row(s) and a one-tailed

<method>_p row, plus a final k row. The p rows print to fixed decimals via the p_terms attribute.

Arguments

p: Numeric vector of one-tailed p-values, each in (0, 1), oriented so that small values support the common directional hypothesis.
method: Character vector naming the methods to compute: any of "fisher", "edgington", "stouffer", "stouffer_weighted"; the default computes all four (the "stouffer_weighted" row appears only when weights is supplied).
weights: Optional non-negative weights for "stouffer_weighted", one per study; degrees of freedom are the conventional choice (Mosteller & Bush, 1954).

Author

Ken Kelley [email protected]

Details

Fisher's statistic is \(-2 \sum \log p_i\), distributed chi square with \(2k\) degrees of freedom under the joint null. Edgington's statistic is the plain sum \(\sum p_i\), referred to a normal approximation with mean \(k/2\) and variance \(k/12\) (accurate for \(k \ge 10\); for smaller \(k\) it is conservative in the tails). The Stouffer statistic is \(\sum z_i / \sqrt{k}\) with \(z_i = \Phi^{-1}(1 - p_i)\), and the weighted variant is \(\sum w_i z_i / \sqrt{\sum w_i^2}\). All four are reported with one-tailed combined p-values, matching the directional inputs.

Methods can disagree, and the disagreement is informative: Rosenthal (1978) notes there is no uniformly best test. In the published analysis, Raudenbush (1984) found three of the four rejecting the null at the .05 level while the df-weighted variant did not, an early warning that large studies were finding smaller effects. Computed from the study-level p-values as tabled, the example below shows two of the four rejecting: Fisher's (\(p = .004\)) and Stouffer's (\(p = .014\)) tests reject, Edgington's sits just above the level (\(p = .051\); the tabled values sum to 7.00 where the paper's Table 2, p. 90, prints a sum of 6.84 with \(p = .04\)), and the df-weighted variant is not close (\(p = .192\)).

References

Edgington, E. S. (1972). An additive method for combining probability values from independent experiments. The Journal of Psychology, 80(2), 351--363.

Fisher, R. A. (1938). Statistical methods for research workers (7th ed.). Oliver & Boyd.

Mosteller, F., & Bush, R. R. (1954). Selected quantitative techniques. In G. Lindzey (Ed.), Handbook of social psychology (Vol. 1). Addison-Wesley.

Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76(1), 85--97.

Rosenthal, R. (1978). Combining results of independent studies. Psychological Bulletin, 85(1), 185--193.

Examples

Run this code

# Raudenbush (1984), Table 2: the four combined tests over the 18
# teacher expectancy studies (Pellegrini & Hicks at its study-level
# values), weighting the Z method by degrees of freedom.
data(teacher_expectancy)
study <- teacher_expectancy[-c(4, 5), ]
p18  <- append(study$p_one_tailed, .010, after = 3)
df18 <- append(study$n_experimental + study$n_control - 2, 42, after = 3)
combine_p(p18, weights = df18)
# Fisher chi square 62.17 on 36 df; Edgington sum near 7; Stouffer
# z near 2.2; and the df-weighted z under 1: the large studies disagree.

Run the code above in your browser using DataLab