caret (version 4.39)

diff.resamples: Inferential Assessments About Model Performance

Description

Methods for making inferences about differences between models

Usage

## S3 method for class 'resamples':
diff(x, models = x$models, metric = x$metrics, test = t.test, ...)
## S3 method for class 'diff.resamples':
summary(object, digits = max(3, getOption("digits") - 3), ...)

Arguments

x
an object generated by resamples
models
a character string for which models to compare
metric
a character string for which metrics to compare
test
a function to compute differences. The output of this function should have scalar outputs called estimate and p.value
object
a object generated by diff.resamples
digits
the number of significant differences to display when printing
...
further arguments to pass to test

Value

  • An object of class "diff.resamples" with elements:
  • callthe call
  • difsa list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows
  • statisticsa list of results generated by test
  • modelsa character string for which models were compared.
  • metricsa character string of performance metrics that were used
  • or...

    An object of class "summary.diff.resamples" with elements:

  • callthe call
  • tablea list of tables that show the differences and p-values

Details

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

See Also

resamples

Examples

Run this code
#load(url("http://caret.r-forge.r-project.org/Classification_and_Regression_Training_files/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))

difs <- diff(resamps)

difs

summary(difs)

Run the code above in your browser using DataCamp Workspace