resamples
Collation and Visualization of Resampling Results
These functions provide methods for collection, analyzing and visualizing a set of resampling results from a common data set.
- Keywords
- models
Usage
resamples(x, ...)## S3 method for class 'default':
resamples(x, modelNames = names(x), ...)
## S3 method for class 'resamples':
summary(object, ...)
Arguments
Details
The ideas and methods here are based on Hothorn et al (2005) and Eugster et al (2008).
The results from train
can have more than one performance metric per resample. Each metric in the input object is saved.
resamples
checks that the resampling results match; that is, the indices in the object trainObject$control$index
are the same. Also, the argument trainControl
returnResamp
should have a value of "final"
for each model.
The summary function computes summary statistics across each model/metric combination.
Value
- An object with class
"resamples"
with elements call the call values a data frame of results where rows correspond to resampled data sets and columns indicate the model and metric models a character string of model labels metrics a character string of performance metrics methods a character string of the train
method
argument values for each model
References
Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699
Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30
See Also
train
, trainControl
, diff.resamples
, xyplot.resamples
, densityplot.resamples
, bwplot.resamples
, splom.resamples
Examples
data(BloodBrain)
set.seed(1)
tmp <- createDataPartition(logBBB,
p = .8,
times = 100)
rpartFit <- train(bbbDescr, logBBB,
"rpart",
tuneLength = 16,
trControl = trainControl(
method = "LGOCV", index = tmp))
ctreeFit <- train(bbbDescr, logBBB,
"ctree",
trControl = trainControl(
method = "LGOCV", index = tmp))
earthFit <- train(bbbDescr, logBBB,
"earth",
tuneLength = 20,
trControl = trainControl(
method = "LGOCV", index = tmp))
## or load pre-calculated results using:
## load(url("http://caret.r-forge.r-project.org/Classification_and_Regression_Training_files/exampleModels.RData"))
resamps <- resamples(list(CART = rpartFit,
CondInfTree = ctreeFit,
MARS = earthFit))
resamps
summary(resamps)