mlr (version 2.7)

estimateRelativeOverfitting: Estimate relative overfitting.

Description

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

Usage

estimateRelativeOverfitting(rdesc, measures, task, learner)
## S3 method for class 'ResampleDesc':
estimateRelativeOverfitting(rdesc, measures, task,
  learner)

Value

[data.frame]. Relative overfitting estimate(s), named by measure(s), for each resampling iteration.

Details

Currently only support for classification and regression tasks is implemented.

References

Bradley Efron and Robert Tibshirani; Improvements on Cross-Validation: The .632+ Bootstrap Method, Journal of the American Statistical Association, Vol. 92, No. 438. (Jun., 1997), pp. 548-560.

See Also

Other performance: makeCostMeasure, makeCustomResampledMeasure, makeMeasure, measures, performance

Examples

task = makeClassifTask(data = iris, target = "Species")
rdesc = makeResampleDesc("CV", iters = 2)
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.knn"))
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.lda"))