scmamp (version 0.2.3)

postHocTest: Post hoc tests for multiple comparison analises

Description

This function is a wrapper to run the post hoc tests. It can run both all vs. control and all vs. all post hoc tests.

Usage

postHocTest(data, algorithms = NULL, group.by = NULL, test = "friedman",
  control = NULL, use.rank = FALSE, sum.fun = mean, correct = "finner",
  alpha = 0.05, ...)

Arguments

data

A matrix or data frame containing the results obtained by the algorithms (columns) in each problem (rows). It can contain additional columns, but if any of the column has to be discarderd (not used neither to group the problems nor to be part of the compa

algorithms

Vector with either the names or the indices of the columns that contain the values to be tested. If not provided, the function assumes that all the columns except those indicated in group.by represent the results obtained by an algorithm.

group.by

Vector with either the names or the indices of the columns to be used to group the data. Each group is tested independently. If NULL, all the data is used for a single comparison.

test

Parameter that indicates the statistical test to be used. It can be either a string indicating one of the available test or a function. As a string, it can take the following values:

• 'wilcoxon'- Wilcoxon Signed Rank test, as in Demsa

Value

• In all cases the function returns a list with three elements, the summarization of the data (a row per group), the raw p-values and the corrected p-values. When the data is grouped and all the pairwise comparisons are performed (no control is provided), the p-values are in three dimensional arrays where the last dimension is corresponds to the group. In any other cases the result is a matrix with one or more rows.

  Note that Shaffer and Bergmann and Hommel's correction can only be applied when all the pairwise tests are conducted, due to their assumptions. Moreover, its use when the data is grouped (multiple pairwise comparsions) is not trivial and, thus, it is not possible to use it when the data is grouped.

item

• control
• use.rank
• sum.fun
• correct
• bergmann - Bergman and Hommel's procedure (similar to Shaffer dynamic), as in Garcia and Herrera (2008)
• holland - Holland's procedure, as in Garcia and Herrera (2010)
• finner - Finner's procedure, as in Garcia and Herrera (2010)
• rom - Rom's procedure, as in Garcia and Herrera (2010)
• li - Li's procedure, as in Garcia and Herrera (2010)
• Any of the methods implemented in the p.adjust function. For a list of options, type p.adjust.methods
• alpha
• ...

code

TRUE

itemize

• shaffer- Shaffer's (static) procedure, as in Garcia and Herrera (2008)

emph

in the same order

References

S. Garcia and F. Herrera (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and ata mining: Experimental analysis of power. Information Sciences, 180, 2044-2064.

Garcia S. and Herrera, F. (2008) An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for All Pairwise Comparisons. Journal of Machine Learning Research, 9, 2677-2694.

Kanji, G. K. (2006) 100 Statistical Tests. SAGE Publications Ltd, 3rd edition.

Demsar, J. (2006) Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7, 1-30.

See Also

friedmanPost, friedmanAlignedRanksPost, quadePost, tukeyPost, adjustShaffer, adjustBergmannHommel, adjustHolland, adjustFinner, adjustRom, adjustLi

Examples

# Grouped data, all pairwise
data(data_blum_2015)
res <- postHocTest (data=data.blum.2015, algorithms=c("FrogCOL", "FrogMIS", "FruitFly"),
                    use.rank=TRUE, group.by=c("Size"), test="quade", correct="finner")

# Data summarization
res$summary

# Corrected pvalues for the first group
res$corrected.pval[, , 1]

# Grouped data, all vs. control
res <- postHocTest (data=data.blum.2015, control="max", use.rank=FALSE,
                    group.by=c("Size","Radius"), test="wilcoxon", correct="finner")

# Data summarization
res$summary

# Corrected pvalues
res$corrected.pval

# Not grouped data
data(data_gh_2008)
postHocTest (data=data.gh.2008, test="aligned ranks", correct="bergmann")