These data generating processes (DGPs) are designed to illustrate specific strengths and weaknesses of different feature importance methods like PFI, CFI, and RFI. Each DGP focuses on one primary challenge to make the differences between methods clear.
sim_dgp_correlated(n = 500L, r = 0.9)sim_dgp_mediated(n = 500L)
sim_dgp_confounded(n = 500L, hidden = TRUE)
sim_dgp_interactions(n = 500L)
sim_dgp_independent(n = 500L)
A regression task (mlr3::TaskRegr) with data.table backend.
(integer(1): 500L) Number of observations to generate.
(numeric(1): 0.9) Correlation between x1 and x2. Must be between -1 and 1.
(logical(1): TRUE) Whether to hide the confounder from the returned task.
If FALSE, the confounder is included as a feature, allowing direct adjustment.
If TRUE (default), only the proxy is available, simulating unmeasured confounding.
sim_dgp_correlated(): Correlated features demonstrating PFI's limitations
sim_dgp_mediated(): Mediated effects showing direct vs total importance
sim_dgp_confounded(): Confounding scenario for conditional sampling
sim_dgp_interactions(): Interaction effects between features
sim_dgp_independent(): Independent features baseline scenario
Correlated Features DGP: This DGP creates highly correlated predictors where PFI will show artificially low importance due to redundancy, while CFI will correctly identify each feature's conditional contribution.
Mathematical Model: $$(X_1, X_2)^T \sim \text{MVN}(0, \Sigma)$$ where \(\Sigma\) is a \(2 \times 2\) covariance matrix with 1 on the diagonal and correlation \(r\) on the off-diagonal. $$X_3 \sim N(0,1), \quad X_4 \sim N(0,1)$$ $$Y = 2 \cdot X_1 + X_3 + \varepsilon$$ where \(\varepsilon \sim N(0, 0.2^2)\).
Feature Properties:
x1: Standard normal from MVN, direct causal effect on y (\(\beta = 2.0\))
x2: Correlated with x1 (correlation = r), NO causal effect on y (\(\beta = 0\))
x3: Independent standard normal, direct causal effect on y (\(\beta = 1.0\))
x4: Independent standard normal, no effect on y (\(\beta = 0\))
Expected Behavior:
Will depend on the used learner and the strength of correlation (r)
Marginal methods (PFI, Marginal SAGE): Should falsely assign importance to x2 due to correlation with x1
CFI Should correctly assign near-zero importance to x2
x2 is a "spurious predictor" - correlated with causal feature but not causal itself
Mediated Effects DGP: This DGP demonstrates the difference between total and direct causal effects. Some features affect the outcome only through mediators.
Mathematical Model: $$\text{exposure} \sim N(0,1), \quad \text{direct} \sim N(0,1)$$ $$\text{mediator} = 0.8 \cdot \text{exposure} + 0.6 \cdot \text{direct} + \varepsilon_m$$ $$Y = 1.5 \cdot \text{mediator} + 0.5 \cdot \text{direct} + \varepsilon$$ where \(\varepsilon_m \sim N(0, 0.3^2)\) and \(\varepsilon \sim N(0, 0.2^2)\).
Feature Properties:
exposure: Has no direct effect on y, only through mediator (total effect = 1.2)
mediator: Mediates the effect of exposure on y
direct: Has both direct effect on y and effect on mediator
noise: No causal relationship to y
Causal Structure: exposure -> mediator -> y <- direct -> mediator
Confounding DGP: This DGP includes a confounder that affects both a feature and the outcome. Uses simple coefficients for easy interpretation.
Mathematical Model: $$H \sim N(0,1)$$ $$X_1 = H + \varepsilon_1$$ $$\text{proxy} = H + \varepsilon_p, \quad \text{independent} \sim N(0,1)$$ $$Y = H + X_1 + \text{independent} + \varepsilon$$ where all \(\varepsilon \sim N(0, 0.5^2)\) independently.
Model Structure:
Confounder H ~ N(0,1) (potentially unobserved)
x1 = H + noise (affected by confounder)
proxy = H + noise (noisy measurement of confounder)
independent ~ N(0,1) (truly independent)
y = H + x1 + independent + noise
Expected Behavior:
PFI: Will show inflated importance for x1 due to confounding
CFI: Should partially account for confounding through conditional sampling and reduce its importance
RFI conditioning on proxy: Should reduce confounding bias by conditioning on proxy
Interaction Effects DGP: This DGP demonstrates a pure interaction effect where features have no main effects.
Mathematical Model: $$Y = 2 \cdot X_1 \cdot X_2 + X_3 + \varepsilon$$ where \(X_j \sim N(0,1)\) independently and \(\varepsilon \sim N(0, 0.5^2)\).
Feature Properties:
x1, x2: Independent features with ONLY interaction effect (no main effects)
x3: Independent feature with main effect only
noise1, noise2: No causal effects
Expected Behavior:
Will depend on the used learner and its ability to model interactions
Independent Features DGP: This is a baseline scenario where all features are independent and their effects are additive. All importance methods should give similar results.
Mathematical Model: $$Y = 2.0 \cdot X_1 + 1.0 \cdot X_2 + 0.5 \cdot X_3 + \varepsilon$$ where \(X_j \sim N(0,1)\) independently and \(\varepsilon \sim N(0, 0.2^2)\).
Feature Properties:
important1-3: Independent features with different effect sizes
unimportant1-2: Independent noise features with no effect
Expected Behavior:
All methods: Should rank features consistently by their true effect sizes
Ground truth: important1 > important2 > important3 > unimportant1,2 (approximately 0)
Ewald F, Bothmann L, Wright M, Bischl B, Casalicchio G, König G (2024). “A Guide to Feature Importance Methods for Scientific Inference.” In Longo L, Lapuschkin S, Seifert C (eds.), Explainable Artificial Intelligence, 440--464. ISBN 978-3-031-63797-1, tools:::Rd_expr_doi("10.1007/978-3-031-63797-1_22").
Other simulation:
sim_dgp_ewald()
Other simulation:
sim_dgp_ewald()
Other simulation:
sim_dgp_ewald()
Other simulation:
sim_dgp_ewald()
Other simulation:
sim_dgp_ewald()
task = sim_dgp_correlated(200)
task$data()
# With different correlation
task_high_cor = sim_dgp_correlated(200, r = 0.95)
cor(task_high_cor$data()$x1, task_high_cor$data()$x2)
task = sim_dgp_mediated(200)
task$data()
# Hidden confounder scenario (traditional)
task_hidden = sim_dgp_confounded(200, hidden = TRUE)
task_hidden$feature_names # proxy available but not confounder
# Observable confounder scenario
task_observed = sim_dgp_confounded(200, hidden = FALSE)
task_observed$feature_names # both confounder and proxy available
task = sim_dgp_interactions(200)
task$data()
task = sim_dgp_independent(200)
task$data()
Run the code above in your browser using DataLab