# diff.resamples

##### Inferential Assessments About Model Performance

Methods for making inferences about differences between models

- Keywords
- models

##### Usage

```
## S3 method for class 'resamples':
diff(x, models = x$models, metric = x$metrics,
test = t.test,
confLevel = 0.95, adjustment = "bonferroni",
...)
```## S3 method for class 'diff.resamples':
summary(object, digits = max(3, getOption("digits") - 3), ...)

##### Arguments

- x
- an object generated by
`resamples`

- models
- a character string for which models to compare
- metric
- a character string for which metrics to compare
- test
- a function to compute differences. The output of this function should have scalar outputs called
`estimate`

and`p.value`

- object
- a object generated by
`diff.resamples`

- adjustment
- any p-value adjustment method to pass to
`p.adjust`

. - confLevel
- confidence level to use for
`dotplot.diff.resamples`

. See Details below. - digits
- the number of significant differences to display when printing
- ...
- further arguments to pass to
`test`

##### Details

The ideas and methods here are based on Hothorn et al (2005) and Eugster et al (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from `confLevel`

to `1-((1-confLevel)/p)`

here `p`

is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

##### Value

- An object of class
`"diff.resamples"`

with elements: call the call difs a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows statistics a list of results generated by `test`

adjustment the p-value adjustment used models a character string for which models were compared. metrics a character string of performance metrics that were used - or...
An object of class

`"summary.diff.resamples"`

with elements: call the call table a list of tables that show the differences and p-values

##### References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

##### See Also

`resamples`

, `dotplot.diff.resamples`

, `densityplot.diff.resamples`

, `bwplot.diff.resamples`

, `levelplot.diff.resamples`

##### Examples

```
#load(url("http://caret.r-forge.r-project.org/exampleModels.RData"))
resamps <- resamples(list(CART = rpartFit,
CondInfTree = ctreeFit,
MARS = earthFit))
difs <- diff(resamps)
difs
summary(difs)
```

*Documentation reproduced from package caret, version 5.15-052, License: GPL-2*