diff.resamples

0th

Percentile

Inferential Assessments About Model Performance

Methods for making inferences about differences between models

Keywords
models
Usage
## S3 method for class 'resamples':
diff(x, models = x$models, metric = x$metrics, 
     test = t.test, 
     confLevel = 0.95, adjustment = "bonferroni",
     ...)

## S3 method for class 'diff.resamples': summary(object, digits = max(3, getOption("digits") - 3), ...)

Arguments
x
an object generated by resamples
models
a character string for which models to compare
metric
a character string for which metrics to compare
test
a function to compute differences. The output of this function should have scalar outputs called estimate and p.value
object
a object generated by diff.resamples
adjustment
any p-value adjustment method to pass to p.adjust.
confLevel
confidence level to use for dotplot.diff.resamples. See Details below.
digits
the number of significant differences to display when printing
...
further arguments to pass to test
Details

The ideas and methods here are based on Hothorn et al (2005) and Eugster et al (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from confLevel to 1-((1-confLevel)/p) here p is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

Value

  • An object of class "diff.resamples" with elements:
  • callthe call
  • difsa list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows
  • statisticsa list of results generated by test
  • adjustmentthe p-value adjustment used
  • modelsa character string for which models were compared.
  • metricsa character string of performance metrics that were used
  • or...

    An object of class "summary.diff.resamples" with elements:

  • callthe call
  • tablea list of tables that show the differences and p-values

References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

See Also

resamples, dotplot.diff.resamples, densityplot.diff.resamples, bwplot.diff.resamples, levelplot.diff.resamples

Aliases
  • diff.resamples
  • summary.diff.resamples
Examples
#load(url("http://caret.r-forge.r-project.org/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))

difs <- diff(resamps)

difs

summary(difs)
Documentation reproduced from package caret, version 5.15-052, License: GPL-2

Community examples

Looks like there are no examples yet.