EMMEANS: Simple-effect analysis and post-hoc multiple comparison.

Description

Easily perform (1) simple-effect and simple-simple-effect analyses, including both simple main effects and simple interaction effects, and (2) post-hoc multiple comparisons (e.g., pairwise, sequential, polynomial), with p-value adjustment for factors with >= 3 levels (using methods such as Bonferroni, Tukey's HSD, and FDR).

Usage

EMMEANS(
  model,
  effect = NULL,
  by = NULL,
  contrast = "pairwise",
  p.adjust = "bonferroni",
  cohen.d = "accurate",
  nsmall = 2,
  sd.pooled = NULL,
  reverse = TRUE
)

Arguments

model

A model fitted by MANOVA or afex::aov_ez.

effect

The effect(s) you want to test. If set to a character string (e.g., "A"), it will output the results of omnibus tests or simple main effects. If set to a character vector (e.g., c("A", "B")), it will also output the results of simple interaction effects.

Moderator variable(s). Default is NULL.

contrast

Contrast method for multiple comparisons. Default is "pairwise".

Alternatives can be "pairwise" ("revpairwise"), "seq" ("consec"), "poly", "eff". For details, see emmeans::contrast-methods.

p.adjust

Adjustment method (of p values) for multiple comparisons. Default is "bonferroni". For polynomial contrasts, default is "none".

Alternatives can be "none"; "fdr", "hochberg", "hommel", "holm"; "tukey", "mvt", "dunnettx", "sidak", "scheffe", "bonferroni". The latter six methods are recommended! For details, see stats::p.adjust and emmeans::confint.emmGrid.

cohen.d

Method to compute Cohen's d in multiple comparisons. Default is "accurate", which will give the most reasonable estimates of Cohen's d and its 95% CI. This method divides the raw means and CIs by the pooled SD corresponding to the effect term (SD_pooled = sqrt(MSE), where MSE is extracted from the ANOVA table).

One alternative can be "eff_size", which uses the eff_size function in the emmeans package. Its point estimates of Cohen's d replicate those by the "accurate" method. However, its CI estimates seem a little bit confusing. For details about this method, see Comparisons and contrasts.

nsmall

Number of decimal places of output. Default is 2.

sd.pooled

By default, it will use sqrt(MSE) to compute Cohen's d. Yet, users can manually set the SD_pooled (e.g., the SD of a reference group).

reverse

The order of levels to be contrasted. Default is TRUE ("higher level vs. lower level").

Value

A result object returned by afex::aov_ez (the same as MANOVA for recursive use).

Details

This function is based on and extends the (1) joint_tests, (2) emmeans, and (3) contrast functions in the R package emmeans. You only need to specify the model object, to-be-tested effect(s), and moderator(s). Then, almost all the outputs you need will be displayed in an elegant manner, including effect sizes (partial \(\eta^2\) and Cohen's d) and their confidence intervals (CIs). 90% CIs for partial \(\eta^2\) and 95% CIs for Cohen's d are reported.

Statistical Details:

Some may confuse the statistical terms "simple effects", "post-hoc tests", and "multiple comparisons". Such a confusion is not uncommon. Here, I explain what these terms actually refer to.

1. Simple Effect

When we speak of "simple effect", we are referring to ...

simple main effect
simple interaction effect (only for designs with 3 or more factors)
simple simple effect (only for designs with 3 or more factors)

When the interaction effect in ANOVA is significant, we should then perform a "simple-effect analysis". In ANOVA, we call it "simple-effect analysis"; in regression, we also call it "simple-slope analysis". They are identical in statistical principles. Nonetheless, the situations in ANOVA can be a bit more complex because we sometimes have a three-factors design.

In a regular two-factors design, we only test "simple main effects". That is, on the different levels of a factor "B", the main effects of "A" would be different. However, in a three-factors (or more) design, we may also test "simple interaction effects" and "simple simple effects". That is, on the different combinations of levels of factors "B" and "C", the main effects of "A" would be different.

In SPSS, we usually use the MANOVA and/or the GLM + /EMMEANS syntax to perform such analyses. Tutorials (in Chinese) for the SPSS syntax can be found in: Tutorial #1, Tutorial #2, and Tutorial #3.

Here, the R function EMMEANS can do the same thing as in SPSS and can do much better and easier (just see the section "Examples"). The outputs include tidy tables as well as the estimates for effect sizes (partial \(\eta^2\)) and their 90% CIs.

To note, simple effects per se do NOT need any form of p-value adjustment, because what we test in simple-effect analyses are still "omnibus F-tests".

2. Post-Hoc Test

The term "post-hoc" means that the tests are performed after ANOVA. Given this, some may (wrongly) regard simple-effect analyses also as a kind of post-hoc tests. However, these two terms should be distinguished. In many situations and softwares, "post-hoc tests" only refer to "post-hoc comparisons" using t-tests and some p-value adjustment techniques. We need post-hoc comparisons only when there are factors with 3 or more levels. For example, we can perform the post-hoc comparisons of mean values (1) across multiple levels of one factor in a pairwise way or (2) particularly between the two conditions "A1B1" and "A2B2".

Post-hoc tests are totally independent of whether there is a significant interaction effect. It only deals with factors with multiple levels. In most cases, we use pairwise comparisons to do post-hoc tests. See the next part for details.

3. Multiple Comparison

As mentioned above, multiple comparisons are post-hoc tests by its nature but do NOT have any relationship with simple-effect analyses. In other words, "(post-hoc) multiple comparisons" are independent of "interaction effects" and "simple effects". What's more, when the simple main effect is of a factor with 3 or more levels, we also need to do multiple comparisons (e.g., pairwise comparisons) within the simple-effect analysis. In this situation (i.e., >= 3 levels), we need p-value adjustment methods such as Bonferroni, Tukey's HSD (honest significant difference), FDR (false discovery rate), and so forth.

There are many ways to do multiple comparisons. All these methods are included in the current EMMEANS function. If you are familiar with SPSS syntax, you may feel that the current R functions MANOVA and EMMEANS are a nice combination of the SPSS syntax MANOVA and GLM + /EMMEANS. Yes, they are. More importantly, they outperform the SPSS syntax, either for its higher convenience or for its more fruitful outputs.

"pairwise" - Pairwise comparisons (default is "higher level - lower level")
"seq" or "consec" - Consecutive (sequential) comparisons
"poly" - Polynomial contrasts (linear, quadratic, cubic, quartic, ...)
"eff" - Effect contrasts (vs. the grand mean)

Examples

Run this code

# NOT RUN {
#### Between-Subjects Design ####

between.1
MANOVA(data=between.1, dv="SCORE", between="A") %>%
  EMMEANS("A")
MANOVA(data=between.1, dv="SCORE", between="A") %>%
  EMMEANS("A", p.adjust="tukey")
MANOVA(data=between.1, dv="SCORE", between="A") %>%
  EMMEANS("A", contrast="seq")
MANOVA(data=between.1, dv="SCORE", between="A") %>%
  EMMEANS("A", contrast="poly")

between.2
MANOVA(data=between.2, dv="SCORE", between=c("A", "B")) %>%
  EMMEANS("A") %>%
  EMMEANS("A", by="B") %>%
  EMMEANS("B") %>%
  EMMEANS("B", by="A")

between.3
MANOVA(data=between.3, dv="SCORE", between=c("A", "B", "C")) %>%
  EMMEANS("A", by="B") %>%
  EMMEANS(c("A", "B"), by="C") %>%
  EMMEANS("A", by=c("B", "C"))
## just to name a few
## you can test many other combinations of effects


#### Within-Subjects Design ####

within.1
MANOVA(data=within.1, dvs="A1:A4", dvs.pattern="A(.)",
       within="A") %>%
  EMMEANS("A")

within.2
MANOVA(data=within.2, dvs="A1B1:A2B3", dvs.pattern="A(.)B(.)",
       within=c("A", "B")) %>%
  EMMEANS("A", by="B") %>%
  EMMEANS("B", by="A")

within.3
MANOVA(data=within.3, dvs="A1B1C1:A2B2C2", dvs.pattern="A(.)B(.)C(.)",
       within=c("A", "B", "C")) %>%
  EMMEANS("A", by="B") %>%
  EMMEANS(c("A", "B"), by="C") %>%
  EMMEANS("A", by=c("B", "C"))


#### Mixed Design ####

mixed.2_1b1w
MANOVA(data=mixed.2_1b1w, dvs="B1:B3", dvs.pattern="B(.)",
       between="A", within="B", sph.correction="GG") %>%
  EMMEANS("A", by="B") %>%
  EMMEANS("B", by="A")

mixed.3_1b2w
MANOVA(data=mixed.3_1b2w, dvs="B1C1:B2C2", dvs.pattern="B(.)C(.)",
       between="A", within=c("B", "C")) %>%
  EMMEANS("A", by="B") %>%
  EMMEANS(c("A", "B"), by="C") %>%
  EMMEANS("A", by=c("B", "C"))

mixed.3_2b1w
MANOVA(data=mixed.3_2b1w, dvs="B1:B2", dvs.pattern="B(.)",
       between=c("A", "C"), within="B") %>%
  EMMEANS("A", by="B") %>%
  EMMEANS("A", by="C") %>%
  EMMEANS(c("A", "B"), by="C") %>%
  EMMEANS("B", by=c("A", "C"))


#### Other Examples ####
air=airquality
air$Day.1or2=ifelse(air$Day %% 2 == 1, 1, 2) %>%
  factor(levels=1:2, labels=c("odd", "even"))
MANOVA(data=air, dv="Temp", between=c("Month", "Day.1or2"),
       covariate=c("Solar.R", "Wind")) %>%
  EMMEANS("Month", contrast="seq") %>%
  EMMEANS("Month", by="Day.1or2", contrast="poly")
# }