Provides a single loop of outer cross-validation to evaluate performance of
ensemble models from SuperLearner package.
nestcv.SuperLearner(
y,
x,
filterFUN = NULL,
filter_options = NULL,
weights = NULL,
balance = NULL,
balance_options = NULL,
modifyX = NULL,
modifyX_useY = FALSE,
modifyX_options = NULL,
outer_method = c("cv", "LOOCV"),
n_outer_folds = 10,
outer_folds = NULL,
parallel_mode = NULL,
cv.cores = 1,
final = TRUE,
na.option = "pass",
verbose = TRUE,
...
)An object with S3 class "nestcv.SuperLearner"
the matched call
Predictions on the left-out outer folds
List object of results from each outer fold containing predictions on left-out outer folds, model result and number of filtered predictors at each fold.
vector of number of observations and number of predictors
original response vector
final response vector (post-balancing)
List of indices of outer test folds
Final fitted model on whole data
Column names of filtered predictors entering final model
Summary statistics of filtered predictors
ROC AUC for binary classification where available.
Overall performance summary. Accuracy and balanced accuracy for classification. ROC AUC for binary classification. RMSE for regression.
Response vector
Dataframe or matrix of predictors. Matrix will be coerced to dataframe as this is the default for SuperLearner.
Filter function, e.g. ttest_filter or relieff_filter.
Any function can be provided and is passed y and x. Ideally returns a
numeric vector with indices of filtered predictors. The custom function can
return a character vector of names of the filtered predictors, but this
will not work with the penalty.factor argument in nestcv.glmnet().
List of additional arguments passed to the filter
function specified by filterFUN.
Weights applied to each sample for models which can use
weights. Note weights and balance cannot be used at the same time.
Weights are not applied in filters.
Specifies method for dealing with imbalanced class data.
Current options are "randomsample" or "smote". Not available if
outercv is called with a formula. See randomsample() and smote()
List of additional arguments passed to the balancing function
Character string specifying the name of a function to modify
x. This can be an imputation function for replacing missing values, or a
more complex function which alters or even adds columns to x. The
required return value of this function depends on the modifyX_useY
setting.
Logical value whether the x modifying function makes
use of response training data from y. If FALSE then the modifyX
function simply needs to return a modified x object, which will be
coerced to a dataframe as required by SuperLearner. If TRUE then the
modifyX function must return a model type object on which predict() can
be called, so that train and test partitions of x can be modified
independently.
List of additional arguments passed to the x
modifying function
String of either "cv" or "LOOCV" specifying whether
to do k-fold CV or leave one out CV (LOOCV) for the outer folds
Number of outer CV folds
Optional list containing indices of test folds for outer
CV. If supplied, n_outer_folds is ignored.
Either "mclapply" or "snow". This determines which
parallel backend to use. The default is parallel::mclapply on unix/mac
and snow on windows. snow uses parallelisation via
SuperLearner::snowSuperLearner.
Number of cores for parallel processing of the outer loops.
Logical whether to fit final model.
Character value specifying how NAs are dealt with.
"omit" is equivalent to na.action = na.omit. "omitcol" removes cases
if there are NA in 'y', but columns (predictors) containing NA are
removed from 'x' to preserve cases. Any other value means that NA are
ignored (a message is given).
Logical whether to print messages and show progress
Additional arguments passed to SuperLearner::SuperLearner()
This performs an outer CV on SuperLearner package ensemble models to measure
performance, allowing balancing of imbalanced datasets as well as filtering
of predictors. SuperLearner prefers dataframes as inputs for the predictors.
If x is a matrix it will be coerced to a dataframe and variable names
adjusted by make.names().
Parallelisation of the outer CV folds is available on linux/mac, but not
available on windows. On windows, snowSuperLearner() is called instead, so
that parallelisation is performed across each call to SuperLearner.
SuperLearner::SuperLearner()