diff.resamples: Inferential Assessments About Model Performance

Description

Methods for making inferences about differences between models

Usage

# S3 method for resamples
diff(
  x,
  models = x$models,
  metric = x$metrics,
  test = t.test,
  confLevel = 0.95,
  adjustment = "bonferroni",
  ...
)
# S3 method for diff.resamples
summary(object, digits = max(3, getOption("digits") - 3), ...)
compare_models(a, b, metric = a$metric[1])

Arguments

an object generated by resamples

models

a character string for which models to compare

metric

a character string for which metrics to compare

test

a function to compute differences. The output of this function should have scalar outputs called estimate and p.value

confLevel

confidence level to use for dotplot.diff.resamples. See Details below.

adjustment

any p-value adjustment method to pass to p.adjust.

…

further arguments to pass to test

object

a object generated by diff.resamples

digits

the number of significant differences to display when printing

a, b

two objects of class train, sbf or rfe with a common set of resampling indices in the control object.

Value

An object of class "diff.resamples" with elements:

call

the call

difs

a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows

statistics

a list of results generated by test

adjustment

the p-value adjustment used

models

a character string for which models were compared.

metrics

a character string of performance metrics that were used

or...

An object of class "summary.diff.resamples" with elements:

call

the call

table

a list of tables that show the differences and p-values

...or (for compare_models) an object of class htest resulting from t.test.

Details

The ideas and methods here are based on Hothorn et al. (2005) and Eugster et al. (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from confLevel to 1-((1-confLevel)/p) here p is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

compare_models is a shorthand function to compare two models using a single metric. It returns the results of t.test on the differences.

References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

Examples

Run this code

# NOT RUN {
# }
# NOT RUN {
#load(url("http://topepo.github.io/caret/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))

difs <- diff(resamps)

difs

summary(difs)

compare_models(rpartFit, ctreeFit)
# }
# NOT RUN {
# }

Run the code above in your browser using DataCamp Workspace