scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems

This is a simple R package aimed at simplifying the statistical analysis of the results in the comparison of algorithms in different problems.

It is mainly focused on non parametric methods and implements Shaffer static and Bergmann and Hommel dynamic corrections for pairwise tests.

The package also includes some plotting tools, such as the critical difference plots shown in Demšar, J., 2006. Indeed, the package is mainly based on the papers:

Demšar, J. (2006) Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7, 1-30.

García S. and Herrera, F. (2008) An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons. Journal of Machine Learning Research, 9, 2677-2694.

García S. and Herrera, F. (2010) Advanced Nonparametric Tests for Multiple Comparison in the Design of Experiments in Computational Intelligence and Data Mining: Experimental Analysis of Power. Information Sciences, 180, 2044-2064.

Package installation

The last version of the package can be installed running the following commands:

if (!require("devtools")) {
  install.packages("devtools")
}

devtools::install_github("b0rxa/scmamp")

The package can also be installed using the tar.gz files in the root directory. First, download the scmamp_*.tar.gz file and install it running:

install.packages(path.tar.gz.file, reps=NULL)

where path.tar.gz.file refers to the path of the downloaded file. Note that these files may not be up to date.

Documentation

The package includes two vignettes, one for the basic use and another with information about the data manipulation. To access to the local versions of these vignettes (once the package is installed):

library("scmamp")
browseVignettes("scmamp")