Learn R Programming
ale (version 0.5.2)

ModelBoot: Statistics and ALE data for a bootstrapped model

Description

A ModelBoot S7 object contains full-model bootstrapped statistics and ALE data for a trained model. Full-model bootstrapping (as distinct from data-only bootstrapping) retrains a model for each bootstrap iteration. Thus, it can be rather slow, though it is much more reliable. However, for obtaining bootstrapped ALE data, plots, and statistics, full-model bootstrapping as provided by ModelBoot is only necessary for models that have not been developed by cross-validation. For cross-validated models, it is sufficient (and much faster) to create a regular [ALE()] object with bootstrapping by setting the boot_it argument in its constructor. In fact, full-model bootstrapping with ModelBoot is often infeasible for slow machine-learning models trained on large datasets, which should rather be cross-validated to assure their reliability. However, for models that have not been cross-validated, full-model bootstrapping with ModelBoot is necessary for reliable results. Further details follow below; see also vignette('ale-statistics').

Usage

ModelBoot(
  model,
  data = NULL,
  ...,
  model_call_string = NULL,
  model_call_string_vars = character(),
  parallel = "all",
  model_packages = NULL,
  y_col = NULL,
  positive = TRUE,
  pred_fun = function(object, newdata, type = pred_type) {
     stats::predict(object =
    object, newdata = newdata, type = type)
 },
  pred_type = "response",
  boot_it = 100,
  boot_alpha = 0.05,
  boot_centre = "mean",
  seed = 0,
  output_model_stats = TRUE,
  output_model_coefs = TRUE,
  output_ale = TRUE,
  output_boot_data = FALSE,
  ale_options = list(),
  ale_p = "auto",
  tidy_options = list(),
  glance_options = list(),
  silent = FALSE
)

Value

An object of class ALE with properties model_stats, model_coefs, ale, model_stats, boot_data, and params.

Arguments

model

Required. See documentation for ALE()

data

dataframe. Dataset to be bootstrapped. This must be the same data on which the model was trained. If not provided, ModelBoot() will try to detect it automatically. For non-standard models, data should be provided.

...

not used. Inserted to require explicit naming of subsequent arguments.

model_call_string

character(1). If NULL (default), the ModelBoot tries to automatically detect and construct the call for bootstrapped datasets. If it cannot, the function will fail early. In that case, a character string of the full call for the model must be provided that includes boot_data as the data argument for the call. See examples.

model_call_string_vars

character. Names of variables included in model_call_string that are not columns in data. If any such variables exist, they must be specified here or else parallel processing may produce an error. If parallelization is disabled with parallel = 0, then this is not a concern. See documentation for the model_packages argument in ALE().

parallel, model_packages

See documentation for ALE()

y_col, pred_fun, pred_type

See documentation for ALE(). Used to calculate bootstrapped performance measures. If left at their default values, then the relevant performance measures are calculated only if these arguments can be automatically detected. Otherwise, they should be specified.

positive

any single atomic value. If the model represented by model or model_call_string is a binary classification model, positive specifies the 'positive' value of y_col (the target outcome), that is, the value of interest that is considered TRUE; any other value of y_col is considered FALSE. This argument is ignored if the model is not a binary classification model. For example, if 2 means TRUE and 1 means FALSE, then set positive = 2.

boot_it

non-negative integer(1). Number of bootstrap iterations for full-model bootstrapping. For bootstrapping of ALE values, see details to verify if ALE() with bootstrapping is not more appropriate than ModelBoot(). If boot_it = 0, then the model is run as normal once on the full data with no bootstrapping.

boot_alpha

numeric(1) from 0 to 1. Alpha for percentile-based confidence interval range for the bootstrap intervals; the bootstrap confidence intervals will be the lowest and highest (1 - 0.05) / 2 percentiles. For example, if boot_alpha = 0.05 (default), the intervals will be from the 2.5 and 97.5 percentiles.

boot_centre

character(1) in c('mean', 'median'). When bootstrapping, the main estimate for the ALE y value is considered to be boot_centre. Regardless of the value specified here, both the mean and median will be available.

seed

integer. Random seed. Supply this between runs to assure identical bootstrap samples are generated each time on the same data. See documentation for ALE() for further details.

output_model_stats

logical(1). If TRUE (default), return overall model statistics using broom::glance() (if available for model) and bootstrap-validated statistics if boot_it > 0.

output_model_coefs

logical(1). If TRUE (default), return model coefficients using broom::tidy() (if available for model).

output_ale

logical(1). If TRUE (default), return ALE data and statistics.

output_boot_data

logical(1). If TRUE, return the full raw data for each bootstrap iteration, specifically, the bootstrapped models and the model row indices. Default FALSE does not return this large, detailed data.

ale_options, tidy_options, glance_options

list of named arguments. Arguments to pass to the ALE() constructor when ale = TRUE, broom::tidy() when model_coefs = TRUE, or broom::glance() when model_stats = TRUE, respectively, beyond (or overriding) their defaults. Note: to obtain p-values for ALE statistics, see the ale_p argument.

ale_p

Same as the p_values argument for the ALE() constructor; see documentation there. This argument overrides the p_values element of the ale_options argument.

silent

See documentation for ALE()

Properties

model_stats

tibble of bootstrapped results from broom::glance(). NULL if model_stats argument is FALSE. In general, only broom::glance() results that make sense when bootstrapped are included, such as df and adj.r.squared. Results that are incomparable across bootstrapped datasets (such as aic) are excluded. In addition, certain model performance measures are included; these are bootstrap-validated with the .632 correction (Efron & Tibshirani 1986) (NOT the .632+ correction):

  • For regression (numeric prediction) models:

    • mae: mean absolute error (MAE)

    • sa_mae: standardized accuracy of the MAE referenced on the mean absolute deviation

    • rmse: root mean squared error (RMSE)

    • sa_rmse: standardized accuracy of the RMSE referenced on the standard deviation

  • For binary or categorical classification (probability) models:

    • auc: area under the ROC curve

model_coefs

A tibble of bootstrapped results from broom::tidy(). NULL if model_coefs argument is FALSE.

ale

A list of bootstrapped ALE results using default ALE() settings unless if overridden with ale_options. NULL if ale argument is FALSE. Elements are:

  * `single`: an `ALE` object of ALE calculations on the full dataset without bootstrapping.
  * `boot`: a list of bootstrapped ALE data and statistics. This element is not an `ALE` object; it uses a special internal format.

boot_data

A tibble of bootstrap results. Each row represents a bootstrap iteration. NULL if boot_data argument is FALSE. The columns are:

  * `it`: the specific bootstrap iteration from 0 to `boot_it` iterations. Iteration 0 is the results from the full dataset (not bootstrapped).
  * `row_idxs`: the row indexes for the bootstrapped sample for that iteration. To save space, the row indexes are returned rather than the full datasets. So, for example, iteration i's bootstrap sample can be reproduced by `data[ModelBoot_obj@boot_data$row_idxs[[2]], ]` where `data` is the dataset and `ModelBoot_obj` is the result of `ModelBoot()`.
  * `model`: the model object trained on that iteration.
  * `ale`: the results of `ALE()` on that iteration.
  * `tidy`: the results of `broom::tidy(model)` on that iteration.
  * `stats`: the results of `broom::glance(model)` on that iteration.
  * `perf`: performance measures on the entire dataset. These are the measures specified above for regression and classification models.

params

Parameters used to calculate bootstrapped data. Most of these repeat the arguments passed to ModelBoot(). These are either the values provided by the user or used by default if the user did not change them but the following additional objects created internally are also provided:

* `y_cats`: same as `ALE@params$y_cats` (see documentation there).
* `y_type`: same as `ALE@params$y_type` (see documentation there).
* `model`: same as `ALE@params$model` (see documentation there).
* `data`: same as `ALE@params$data` (see documentation there).

Full-model bootstrapping

No modelling results, with or without ALE, should be considered reliable without appropriate validation. For ALE, both the trained model itself and the ALE that explains the trained model must be validated. ALE must be validated by bootstrapping. The trained model might be validated either by cross-validation or by bootstrapping. For ALE that explains trained models that have been developed by cross-validation, it is sufficient to bootstrap just the training data. That is what the ALE object does with its boot_it argument. However, unvalidated models must be validated by bootstrapping them along with the calculation of ALE; this is what the ModelBoot object does with its boot_it argument.

ModelBoot() carries out full-model bootstrapping to validate models. Specifically, it:

  • Creates multiple bootstrap samples (default 100; the user can specify any number);

  • Creates a model on each bootstrap sample;

  • Calculates overall model statistics, variable coefficients, and ALE values for each model on each bootstrap sample;

  • Calculates the mean, median, and lower and upper confidence intervals for each of those values across all bootstrap samples.

References

Okoli, Chitu. 2023. “Statistical Inference Using Machine Learning and Classical Techniques Based on Accumulated Local Effects (ALE).” arXiv. doi:10.48550/arXiv.2310.09877.<

Efron, Bradley, and Robert Tibshirani. "Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy." Statistical science (1986): 54-75. doi:10.1214/ss/1177013815

Examples

Run this code

# attitude dataset
attitude

## ALE for generalized additive models (GAM)
## GAM is tweaked to work on the small dataset.
gam_attitude <- mgcv::gam(rating ~ complaints + privileges + s(learning) +
                            raises + s(critical) + advance,
                          data = attitude)
summary(gam_attitude)

# \donttest{
# Full model bootstrapping

# # To generate the code, uncomment the following lines.
# # For speed, this vignette loads a pre-created ModelBoot object.
# # For standard models like lm that store their data,
# # there is no need to specify the data argument.
# # 100 bootstrap iterations by default.
# mb_gam_attitude <- ModelBoot(gam_attitude)
# saveRDS(mb_gam_attitude, file.choose())
mb_gam_attitude <- url(paste0(
  'https://github.com/tripartio/ale/raw/main/download/',
  'mb_gam_attitude.0.5.2.rds'
  )) |>
  readRDS()

# If the model is not standard, supply model_call_string with 'data = boot_data'
# in the string instead of the actual dataset name (in addition to the actual dataset
# as the 'data' argument directly to the `ModelBoot` constructor).
# mb_gam_attitude <- ModelBoot(
#   gam_attitude,
#   data = attitude,  # the actual dataset
#   model_call_string = 'mgcv::gam(
#     rating ~ complaints + privileges + s(learning) +
#       raises + s(critical) + advance,
#     data = boot_data  # required for model_call_string
#   )'
# )

# Model statistics and coefficients
mb_gam_attitude@model_stats
mb_gam_attitude@model_coefs

# Plot ALE
plot(mb_gam_attitude)

# Retrieve ALE data
get(mb_gam_attitude, type = 'boot')    # bootstrapped
get(mb_gam_attitude, type = 'single')  # full (unbootstrapped) model
# See get.ALE() for other options

# }

Run the code above in your browser using DataLab