# diff.resamples

##### Inferential Assessments About Model Performance

Methods for making inferences about differences between models

- Keywords
- models

##### Usage

```
# S3 method for resamples
diff(
x,
models = x$models,
metric = x$metrics,
test = t.test,
confLevel = 0.95,
adjustment = "bonferroni",
...
)
```# S3 method for diff.resamples
summary(object, digits = max(3, getOption("digits") - 3), ...)

compare_models(a, b, metric = a$metric[1])

##### Arguments

- x
an object generated by

`resamples`

- models
a character string for which models to compare

- metric
a character string for which metrics to compare

- test
a function to compute differences. The output of this function should have scalar outputs called

`estimate`

and`p.value`

- confLevel
confidence level to use for

`dotplot.diff.resamples`

. See Details below.- adjustment
any p-value adjustment method to pass to

`p.adjust`

.- …
further arguments to pass to

`test`

- object
a object generated by

`diff.resamples`

- digits
the number of significant differences to display when printing

- a, b
two objects of class

`train`

,`sbf`

or`rfe`

with a common set of resampling indices in the`control`

object.

##### Details

The ideas and methods here are based on Hothorn et al. (2005) and Eugster et al. (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from
`confLevel`

to `1-((1-confLevel)/p)`

here `p`

is the number
of pair-wise comparisons are being made. For other correction methods, no
such change is used.

`compare_models`

is a shorthand function to compare two models using a
single metric. It returns the results of `t.test`

on the
differences.

##### Value

An object of class `"diff.resamples"`

with elements:

the call

a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows

a list of results generated by `test`

the p-value adjustment used

a character string for which models were compared.

a character string of performance metrics that were used

or...

An object of class "summary.diff.resamples" with elements:

the call

a list of tables that show the differences and p-values

...or (for compare_models) an object of class htest resulting from t.test.

##### References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

##### See Also

`resamples`

, `dotplot.diff.resamples`

,
`densityplot.diff.resamples`

,
`bwplot.diff.resamples`

, `levelplot.diff.resamples`

##### Examples

```
# NOT RUN {
# }
# NOT RUN {
#load(url("http://topepo.github.io/caret/exampleModels.RData"))
resamps <- resamples(list(CART = rpartFit,
CondInfTree = ctreeFit,
MARS = earthFit))
difs <- diff(resamps)
difs
summary(difs)
compare_models(rpartFit, ctreeFit)
# }
# NOT RUN {
# }
```

*Documentation reproduced from package caret, version 6.0-86, License: GPL (>= 2)*