caret (version 5.07-001)

resamples: Collation and Visualization of Resampling Results


These functions provide methods for collection, analyzing and visualizing a set of resampling results from a common data set.


resamples(x, ...)

## S3 method for class 'default': resamples(x, modelNames = names(x), ...)

## S3 method for class 'resamples': summary(object, ...)



  • An object with class "resamples" with elements
  • callthe call
  • valuesa data frame of results where rows correspond to resampled data sets and columns indicate the model and metric
  • modelsa character string of model labels
  • metricsa character string of performance metrics
  • methodsa character string of the train method argument values for each model


The ideas and methods here are based on Hothorn et al (2005) and Eugster et al (2008).

The results from train can have more than one performance metric per resample. Each metric in the input object is saved.

resamples checks that the resampling results match; that is, the indices in the object trainObject$control$index are the same. Also, the argument trainControl returnResamp should have a value of "final" for each model.

The summary function computes summary statistics across each model/metric combination.


Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

See Also

train, trainControl, diff.resamples, xyplot.resamples, densityplot.resamples, bwplot.resamples, splom.resamples


tmp <- createDataPartition(logBBB,
                           p = .8,
                           times = 100)

rpartFit <- train(bbbDescr, logBBB,
                  tuneLength = 16,
                  trControl = trainControl(
                    method = "LGOCV", index = tmp))

ctreeFit <- train(bbbDescr, logBBB,
                  trControl = trainControl(
                    method = "LGOCV", index = tmp))

earthFit <- train(bbbDescr, logBBB,
                  tuneLength = 20,
                  trControl = trainControl(
                    method = "LGOCV", index = tmp))

## or load pre-calculated results using:
## load(url(""))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))