diff.resamples
Inferential Assessments About Model Performance
Methods for making inferences about differences between models
 Keywords
 models
Usage
"diff"(x, models = x$models, metric = x$metrics, test = t.test, confLevel = 0.95, adjustment = "bonferroni", ...)
"summary"(object, digits = max(3, getOption("digits")  3), ...)
compare_models(a, b, metric = a$metric[1])
Arguments
 x
 an object generated by
resamples
 models
 a character string for which models to compare
 metric
 a character string for which metrics to compare
 test
 a function to compute differences. The output of this function should have scalar outputs called
estimate
andp.value
 object
 a object generated by
diff.resamples
 adjustment
 any pvalue adjustment method to pass to
p.adjust
.  confLevel
 confidence level to use for
dotplot.diff.resamples
. See Details below.  digits
 the number of significant differences to display when printing
 a, b
 two objects of class
train
,sbf
orrfe
with a common set of resampling indices in thecontrol
object.  ...
 further arguments to pass to
test
Details
The ideas and methods here are based on Hothorn et al. (2005) and Eugster et al. (2008).
For each metric, all pairwise differences are computed and tested to assess if the difference is equal to zero.
When a Bonferroni correction is used, the confidence level is changed from confLevel
to 1((1confLevel)/p)
here p
is the number of pairwise comparisons are being made. For other correction methods, no such change is used.
compare_models
is a shorthand function to compare two models using a single metric. It returns the results of t.test
on the differences.
Value

An object of class
 call
 the call
 difs
 a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows
 statistics
 a list of results generated by
test
 adjustment
 the pvalue adjustment used
 models
 a character string for which models were compared.
 metrics
 a character string of performance metrics that were used or...An object of class
 call
 the call
 table
 a list of tables that show the differences and pvalues ...or (for
"diff.resamples"
with elements:
"summary.diff.resamples"
with elements:
compare_models
) an object of class htest
resulting from t.test
.
References
Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675699
Eugster et al. Exploratory and inferential analysis of benchmark experiments. LudwigsMaximiliansUniversitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30
See Also
resamples
, dotplot.diff.resamples
, densityplot.diff.resamples
, bwplot.diff.resamples
, levelplot.diff.resamples
Examples
## Not run:
# #load(url("http://topepo.github.io/caret/exampleModels.RData"))
#
# resamps < resamples(list(CART = rpartFit,
# CondInfTree = ctreeFit,
# MARS = earthFit))
#
# difs < diff(resamps)
#
# difs
#
# summary(difs)
#
# compare_models(rpartFit, ctreeFit)
# ## End(Not run)