```
## S3 method for class 'resamples':
diff(x, models = x$models, metric = x$metrics,
test = t.test,
confLevel = 0.95, adjustment = "bonferroni",
...)
```## S3 method for class 'diff.resamples':
summary(object, digits = max(3, getOption("digits") - 3), ...)

x

an object generated by

`resamples`

models

a character string for which models to compare

metric

a character string for which metrics to compare

test

a function to compute differences. The output of this function should have scalar outputs called

`estimate`

and `p.value`

object

a object generated by

`diff.resamples`

adjustment

any p-value adjustment method to pass to

`p.adjust`

.confLevel

confidence level to use for

`dotplot.diff.resamples`

. See Details below.digits

the number of significant differences to display when printing

...

further arguments to pass to

`test`

- An object of class
`"diff.resamples"`

with elements: call the call difs a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows statistics a list of results generated by `test`

adjustment the p-value adjustment used models a character string for which models were compared. metrics a character string of performance metrics that were used - or...
An object of class

`"summary.diff.resamples"`

with elements: call the call table a list of tables that show the differences and p-values

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from `confLevel`

to `1-((1-confLevel)/p)`

here `p`

is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

`resamples`

, `dotplot.diff.resamples`

, `densityplot.diff.resamples`

, `bwplot.diff.resamples`

, `levelplot.diff.resamples`

#load(url("http://caret.r-forge.r-project.org/Classification_and_Regression_Training_files/exampleModels.RData")) resamps <- resamples(list(CART = rpartFit, CondInfTree = ctreeFit, MARS = earthFit)) difs <- diff(resamps) difs summary(difs)