Base class generalizing refit-based variable importance measures.
Default corresponds to leaving out each feature n_repeats times, which
corresponds to LOCO (Leave One Covariate Out).
xplainfi::FeatureImportanceMethod -> WVIM
direction(character(1)) Either "leave-out" or "leave-in".
design(logical()) Feature selection design matrix where TRUE equals "left in" and FALSE "left out".
Columns correspond to task$feature_names and the number of rows corresponds to length(features) * n_repeats.
The base matrix is created by wvim_design_matrix and then replicated n_repeats times before.
instance(FSelectInstanceBatchSingleCrit) The mlr3fselect feature selection instance containing
also the archive of all evaluations, possible useful for future use. Only stored if store_instance is TRUE.
new()Creates a new instance of this R6 class.
WVIM$new(
task,
learner,
measure = NULL,
resampling = NULL,
features = NULL,
groups = NULL,
direction = c("leave-out", "leave-in"),
label = "Williamson's Variable Importance Measure (WVIM)",
n_repeats = 30L
)task, learner, measure, resampling, features, groupsPassed to FeatureImportanceMethod for construction.
direction(character(1)) Either "leave-out" or "leave-in".
label(character(1)) Method label.
n_repeats(integer(1): 30L) Number of refit iterations per resampling iteration.
importance()Get aggregated importance scores.
Extends the base $importance() method to support ci_method = "lei".
This implements distribution-free inference based on Lei et al. (2018), testing observation-wise loss differences using the Wilcoxon signed-rank test by default.
Lei et al. (2018) proposed this method specifically for LOCO with L1 (absolute) loss, median aggregation, and a single train/test split (holdout). The inference is conditional on the training data, requiring i.i.d. test observations from a single split. While the aggregation function, statistical test, and resampling strategy are parameterizable, deviating from these defaults may invalidate the theoretical guarantees.
For a comprehensive overview of inference methods, see
vignette("inference", package = "xplainfi").
WVIM$importance(
relation = NULL,
standardize = FALSE,
ci_method = c("none", "raw", "nadeau_bengio", "quantile", "lei"),
conf_level = 0.95,
alternative = c("two.sided", "greater"),
test = c("wilcoxon", "t", "fisher", "binomial"),
B = 1999,
aggregator = NULL,
p_adjust = "none",
...
)relation(character(1)) How to relate perturbed scores to originals ("difference" or "ratio"). If NULL, uses stored parameter value.
standardize(logical(1): FALSE) If TRUE, importances are standardized by the highest score so all scores fall in [-1, 1].
ci_method(character(1): "none") Variance estimation method. In addition to base methods ("none", "raw", "nadeau_bengio", "quantile"),
WVIM methods support "lei" for distribution-free inference (Lei et al., 2018).
conf_level(numeric(1): 0.95) Confidence level to use for confidence interval construction when ci_method != "none".
alternative(character(1): "two.sided") Type of alternative hypothesis for statistical tests.
"greater" tests H0: importance <= 0 vs H1: importance > 0 (one-sided).
"two.sided" tests H0: importance = 0 vs H1: importance != 0.
test(character(1): "wilcoxon") Test to use for Lei et al. inference. One of "wilcoxon", "t", "fisher", or "binomial".
Only used when ci_method = "lei".
B(integer(1): 1999) Number of replications for Fisher test. Only used when ci_method = "lei" and test = "fisher".
aggregator(function: stats::median) Aggregation function for computing the point estimate from observation-wise
importance values. Defaults to stats::median as proposed by Lei et al. (2018). Only used when ci_method = "lei".
p_adjust(character(1): "none") Method for p-value adjustment for multiple comparisons.
Accepts any method supported by stats::p.adjust.methods, e.g. "holm", "bonferroni", "BH", "none".
When "bonferroni", confidence intervals are also adjusted (alpha/k).
For other correction methods (e.g. "holm", "BH"), only p-values are adjusted;
confidence intervals remain at the nominal conf_level because these sequential/adaptive
procedures do not have a clean per-comparison alpha for CI construction.
...Additional arguments passed to the base method.
(data.table) Aggregated importance scores.
compute()Computes leave-out or leave-in feature importance.
wvim_design_matrix(task$feature_names, "leave-out") corresponds to LOCO.
WVIM$compute(
store_models = TRUE,
store_backends = TRUE,
store_instance = FALSE
)store_models, store_backends(logical(1): TRUE) Whether to store fitted models / data backends, passed to mlr3::resample internally
backends in resample result.
Required for some measures, but may increase memory footprint.
store_instance(logical(1): FALSE) Whether to store the mlr3fselect::mlr3fselect instance in $instance.
clone()The objects of this class are cloneable with this method.
WVIM$clone(deep = FALSE)deepWhether to make a deep clone.
Lei J, G'Sell M, Rinaldo A, Tibshirani R, Wasserman L (2018). “Distribution-Free Predictive Inference for Regression.” Journal of the American Statistical Association, 113(523), 1094--1111. tools:::Rd_expr_doi("10.1080/01621459.2017.1307116").