holdOut(sys, ds, sets, itsInfo = F)sys is an object of the class learner representing the
system to evaluate.
ds is an object of the class dataset representing the data
set to be used in the evaluation.
sets is an object of the class cvSettings representing the
cross validation experimental settings to use.
hldRun.
It is the user responsibility to decide which statistics are to be evaluated on each iteration and how they are calculated. This is done by creating a function that the user knows it will be called by this hold out routine at each repetition of the learn+test process. This user-defined function must assume that it will receive in the first 3 arguments a formula, a training set and a testing set, respectively. It should also assume that it may receive any other set of parameters that should be passed towards the learning algorithm. The result of this user-defined function should be a named vector with the values of the statistics to be estimated obtained by the learner when trained with the given training set, and tested on the given test set. See the Examples section below for an example of these functions.
If the itsInfo parameter is set to the value
TRUE then the hldRun object that is the result
of the function will have an attribute named itsInfo
that will contain extra information from the individual repetitions of
the hold out process. This information can be accessed by the user by
using the function attr(),
e.g. attr(returnedObject,'itsInfo'). For this
information to be collected on this attribute the user needs to code
its user-defined functions in a way that it returns the vector of the
evaluation statistics with an associated attribute named
itInfo (note that it is "itInfo" and not "itsInfo" as
above), which should be a list containing whatever information the
user wants to collect on each repetition. This apparently complex
infra-structure allows you to pass whatever information you which from
each iteration of the experimental process. A typical example is the
case where you want to check the individual predictions of the model
on each test case of each repetition. You could pass this vector of
predictions as a component of the list forming the attribute
itInfo of the statistics returned by your user-defined
function. In the end of the experimental process you will be able to
inspect/use these predictions by inspecting the attribute
itsInfo of the hldRun object returned by the
holdOut() function. See the Examples section for an
illustration of this potentiality.
experimentalComparison,
hldRun,hldSettings, monteCarlo, crossValidation, loocv, bootstrap
## Estimating the mean absolute error and the normalized mean squared
## error of rpart on the swiss data, using 10 repetitions of 70%-30%
## Hold Out experiment
data(swiss)
## First the user defined function (note: can have any name)
hld.rpart <- function(form, train, test, ...) {
require(rpart)
model <- rpart(form, train, ...)
preds <- predict(model, test)
regr.eval(resp(form, test), preds,
stats=c('mae','nmse'), train.y=resp(form, train))
}
## Now the evaluation
eval.res <- holdOut(learner('hld.rpart',pars=list()),
dataset(Infant.Mortality ~ ., swiss),
hldSettings(10,0.3,1234))
## Check a summary of the results
summary(eval.res)
## Plot them
## Not run:
# plot(eval.res)
# ## End(Not run)
## An illustration of the use of the itsInfo parameter.
## In this example the goal is to be able to check values predicted on
## each iteration of the experimental process (e.g. checking for extreme
## values)
## We need a different user-defined function that exports this
## information as an attribute
hld.rpart <- function(form, train, test, ...) {
require(rpart)
model <- rpart(form, train, ...)
preds <- predict(model, test)
eval.stats <- regr.eval(resp(form, test), preds,
stats=c('mae','nmse'),
train.y=resp(form,train))
structure(eval.stats,itInfo=list(predictions=preds))
}
## Now lets run the experimental comparison
eval.res <- holdOut(learner('hld.rpart',pars=list()),
dataset(Infant.Mortality ~ ., swiss),
hldSettings(10,0.3,1234),
itsInfo=TRUE)
## getting the information with the predictions for all 10 repetitions
info <- attr(eval.res,'itsInfo')
## checking the predictions on the 5th repetition
info[[5]]
Run the code above in your browser using DataLab