caret (version 5.05.004)

diff.resamples: Inferential Assessments About Model Performance

Description

Methods for making inferences about differences between models

Usage

## S3 method for class 'resamples':
diff(x, models = x$models, metric = x$metrics, 
     test = t.test, 
     confLevel = 0.95, adjustment = "bonferroni",
     ...)
## S3 method for class 'diff.resamples':
summary(object, digits = max(3, getOption("digits") - 3), ...)

Value

• An object of class "diff.resamples" with elements:
• call
  the call
• difs
  a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows
• statistics
  a list of results generated by test
• adjustment
  the p-value adjustment used
• models
  a character string for which models were compared.
• metrics
  a character string of performance metrics that were used
• or...

  An object of class "summary.diff.resamples" with elements:

• call
  the call
• table
  a list of tables that show the differences and p-values

Details

The ideas and methods here are based on Hothorn et al (2005) and Eugster et al (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from confLevel to 1-((1-confLevel)/p) here p is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

See Also

resamples, dotplot.diff.resamples, densityplot.diff.resamples, bwplot.diff.resamples, levelplot.diff.resamples

Examples

#load(url("http://caret.r-forge.r-project.org/Classification_and_Regression_Training_files/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))

difs <- diff(resamps)

difs

summary(difs)

Run the code above in your browser using DataCamp Workspace