Analyzes the sensitivity of the model performance (measured by MAE) to changes in a single parameter. This function bins the parameter range to identify the minimum MAE for each bin, helping to understand how robust the model is to parameter choices.
parameter_sensitivity_analysis(
param,
samples,
bins = 30,
mae_col = "Holdout_MAE",
threshold_pct = 5,
min_samples = 1
)
An object of class "parameter_sensitivity" containing:
Vector of parameter bin midpoints
Vector of minimum MAE values per bin
Name of analyzed parameter
Threshold value (default: min. +5%)
Minimum MAE value across all bins
Number of samples per bin
The character name of the parameter to analyze.
A data frame containing parameter samples and performance metrics.
The integer number of bins to divide the parameter range into.
The character name of the column containing the Mean Absolute Error (MAE) values.
A numeric percentage above the minimum MAE to define an acceptable performance threshold.
The integer minimum number of samples required in a bin for it to be included in the analysis.
The function performs these steps:
Cleans the input data using Median Absolute Deviation (MAD) to remove outliers.
Bins the parameter values into equal-width bins.
Calculates the minimum MAE within each bin to create an empirical performance curve.
Identifies a performance threshold based on a percentage above the global minimum MAE.
Returns an S3 object for plotting and further analysis.