MLwrap (version 0.3.0)

sensitivity_analysis: Perform Sensitivity Analysis and Interpretable ML methods

Description

As the final step in the MLwrap package workflow, this function performs Sensitivity Analysis (SA) on a fitted ML model stored in an analysis_object (in the examples, e.g., tidy_object). It evaluates the importance of features using various methods such as Permutation Feature Importance (PFI), SHAP (SHapley Additive exPlanations), Integrated Gradients, Olden sensitivity analysis, and Sobol indices. The function generates numerical results and visualizations (e.g., bar plots, box plots, beeswarm plots) to help interpret the impact of each feature on the model's predictions for both regression and classification tasks, providing critical insights after model training and evaluation.

Following the steps of data preprocessing, model fitting, and performance assessment in the MLwrap pipeline, sensitivity_analysis() processes the training and test data using the preprocessing recipe stored in the analysis_object, applies the specified SA methods, and stores the results within the analysis_object. It supports different metrics for evaluation and handles multi-class classification by producing class-specific analyses and plots, ensuring a comprehensive understanding of model behavior (Iooss & Lemaître, 2015).

Usage

sensitivity_analysis(
  analysis_object,
  methods = c("PFI"),
  metric = NULL,
  use_test = FALSE
)

Value

An updated analysis_object containing sensitivity analysis results. Results are stored in the sensitivity_analysis slot as a list, with each method's results accessible by name. Generates bar, box, and beeswarm plots for feature importance visualization, completing the workflow with actionable insights.

Arguments

analysis_object

analysis_object created from fine_tuning function.

methods

Method to be used. A string of the method name: "PFI" (Permutation Feature Importance), "SHAP" (SHapley Additive exPlanations), "Integrated Gradients" (Neural Network only), "Olden" (Neural Networks only), "Sobol_Jansen" (only when all input features are continuous), "Friedman H-stat" (Friedman's H-statistics for feature interaction).

metric

Metric used for "PFI" method (Permutation Feature Importance). A string of the name of metric (see Metrics).

use_test

Logical. Compute methods using the test set instead of the training set (default = FALSE).

Details

As the concluding phase of the MLwrap workflow—after data preparation, model training, and evaluation—this function interprets models by quantifying and visualizing feature importance. It validates input with check_args_sensitivity_analysis(), preprocesses data using the recipe stored in analysis_object$transformer, then calculates feature importance via the specified methods:

• PFI (Permutation Feature Importance): Assesses importance by shuffling feature values and measuring the change in model performance (using the specified or default metric).

• SHAP (SHapley Additive exPlanations): Computes SHAP values to explain individual predictions by attributing contributions to each feature.

• Integrated Gradients: Evaluates feature importance by integrating gradients of the model's output with respect to input features.

• Olden: Calculates sensitivity based on connection weights, typically for neural network models, to determine feature contributions.

• Sobol_Jansen: Variance-based global sensitivity analysis that decomposes model output variance into contributions from individual features and their interactions. Quantifies how much each feature accounts for prediction variability. Only for continuous outcomes. Estimates first-order and total-order Sobol indices using the Jansen (1999) Monte Carlo estimator.

• Friedman H-stat: Computes the Friedman H-statistic for each feature. It measures the strength of interaction effects relative to main effects, following the formulation in Interpretable Machine Learning (Christoph Molnar). After ranking features by global H-statistic, the top 5 features are selected and all their pairwise interactions are computed, returning both raw interaction strength and normalized interaction scores (0–1).

For classification tasks with more than two outcome levels, the function generates separate results and plots for each class. Visualizations include bar plots for importance metrics, box plots for distribution of values, and beeswarm plots for detailed feature impact across observations. All results are stored in the analysis_object under the sensitivity_analysis slot, finalizing the MLwrap pipeline with a deep understanding of model drivers.

References

Iooss, B., & Lemaître, P. (2015). A review on global sensitivity analysis methods. In: G. Dellino & C. Meloni (Eds.), Uncertainty Management in Simulation-Optimization of Complex Systems. Operations Research/Computer Science Interfaces Series (vol. 59). Springer, Boston, MA. tools:::Rd_expr_doi("10.1007/978-1-4899-7547-8_5")

Jansen, M. J. W. (1999). Analysis of variance designs for model output. Computer Physics Communications, 117(1-2), 35–43. tools:::Rd_expr_doi("10.1016/S0010-4655(98)00154-4")

Molnar, C. (2022). Interpretable Machine Learning.
https://christophm.github.io/interpretable-ml-book/

Examples

# Example: Using PFI

wrap_object <- preprocessing(
       df = sim_data,
       formula = psych_well ~ depression + life_sat,
       task = "regression"
       )
wrap_object <- build_model(
               analysis_object = wrap_object,
               model_name = "Random Forest",
               hyperparameters = list(
                                 mtry = 2,
                                 trees = 3
                                 )
                           )
set.seed(123) # For reproducibility
wrap_object <- fine_tuning(wrap_object,
                tuner = "Grid Search CV",
                metrics = c("rmse")
                )
wrap_object <- sensitivity_analysis(wrap_object, methods = "PFI")

# Extracting Results

table_pfi <- table_pfi_results(wrap_object)