Impute missing values with prefered model, sequentially, with hyperparametertuning and with PMM (if wanted)
vimpute(
data,
...,
considered_variables = names(data),
method = setNames(as.list(rep("ranger", length(considered_variables))),
considered_variables),
pmm = FALSE,
pmm_k = NULL,
pmm_k_method = "mean",
learner_params = NULL,
formula = FALSE,
makeNA = NULL,
donorcond = NULL,
sequential = TRUE,
nseq = 10,
eps = 0.005,
imp_var = TRUE,
keep_all_columns = TRUE,
pred_history = FALSE,
tune = FALSE,
verbose = FALSE,
boot = NULL,
robustboot = "stratified",
uncert = "pmm",
m = 1L,
seed = NULL,
tuned_params = NULL,
tune_control = NULL,
predictors = NULL,
visit_sequence = "asis",
spec = NULL
)For m = 1: the imputed dataset, classed like the input
(data.frame in, data.frame out; data.table in, data.table out). When
tune = TRUE the tuning report is attached as
attr(result, "tuning_log"); when pred_history = TRUE the
prediction history is attached as attr(result, "pred_history");
sequential runs attach the per-variable convergence matrix as
attr(result, "convergence") and the chain statistics as
attr(result, "chain"); the per-variable model quality (NRMSE/PFC,
out-of-bag for ranger, in-sample otherwise) is attached as
attr(result, "model_error") -- the return is always the data
itself, never a wrapper list.
For m > 1: a vimmi object.
Dataset with missing values. Provide as a data.table.
Optional bare grammar formulas, one per variable:
target ~ predictors | method(...) -- e.g.
Sleep ~ Dream + Span | ranger(tune = TRUE) or
NonD ~ . | robust(donorcond = ">= 0") -- plus an optional
.default = spec for unlisted variables. A plain-column
right-hand side restricts the predictors (works for every
method); a right-hand side with transformations (s(x),
log(x), I(x^2), interactions) becomes a model
formula (formula-capable methods only); . means all other
variables. Grammar formulas compile to spec objects and cannot be
combined with spec = or the flat per-variable arguments.
A character vector of variable names to be either imputed or used as predictors, excluding irrelevant columns from the imputation process. Excluded columns are still returned unchanged by default (see keep_all_columns).
Specifies the imputation method for each variable. Can be provided either:
as a single global method (e.g. "ranger"), applied to all variables, or
as a named list (e.g. list(var1 = "xgboost", var2 = "robust")), assigning a method to each variable individually. Built-in methods:
ranger (Random Forest)
xgboost (Gradient Boosting)
regularized (glmnet regression/classification)
robust (robustbase::lmrob / glmrob)
gam (Generalized Additive Model via mgcv::gam)
robgam (Robust GAM with outlier downweighting, simple or iterative reweighting)
restricted (ECOSolveR least-squares regression with validate rules)
Additional methods backed by any mlr3 learner pair can be added with
register_vimpute_method(); vimpute_methods()
lists everything currently registered.
Predictive Mean Matching (PMM) settings. Can be provided:
as a single TRUE/FALSE (global), or
as a named list, assigning PMM per (numeric) variable.
Number of nearest neighbors used in PMM. Accepted forms:
single global integer (applies to all variables), or
named list assigning values per variable, or
NULL (default), meaning:
k = 1 automatically for variables using PMM,
k = NULL for variables without PMM
Aggregation method used when pmm_k > 1 in PMM.
Default is "mean".
Accepted forms:
single global string ("mean", "median", "random"), or
single global function (called with the k nearest observed values), or
named list assigning methods per variable, or
NULL values inside such lists, which fall back to "mean"
Semantics:
"mean": mean of the k nearest neighbors
"median": median of the k nearest neighbors
"random": random draw of one among the k nearest neighbors
function: custom aggregator returning one numeric value
Hyperparameters for the chosen methods. Can be provided in three ways:
Per variable (e.g. list(mpg = list(num.trees = 500)))
Per method (e.g. list(ranger = list(num.trees = 600)))
Global, applied to all variables using the same method
For restricted, set save_optimization_problem = TRUE to attach the
exact ECOS problem arguments to the returned object under the
restricted_optimization_problems attribute.
Set robust = TRUE to replace least squares with Huber loss; huber_k
controls its tuning constant and defaults to 1.345.
Optional modeling formula to restrict or transform predictor variables. Only supported for methods whose registry entry declares formula support: among the built-ins regularized (glmnet), robust (lmrob/glmrob), gam (mgcv::gam), robgam (robust GAM), and restricted (ECOSolveR least-squares with validate rules) Provide as a named list, e.g.:
list(mpg = mpg ~ hp + drat)
list(hp = log(hp) ~ wt + cyl) For X: follows the rules of model.matrix For Y: transformations supported are log(), exp(), sqrt(), I(1/..). Only applicable for numeric variables.
Optional named list that defines additional values to be treated as imputable missing
values per variable, similar to kNN(). For variables listed in makeNA,
only the specified values are imputed; existing NA values are left untouched.
Variables not listed in makeNA continue to impute regular NA values.
Optional named list of donor conditions per variable, similar to kNN().
Rows whose observed target values do not satisfy the condition are excluded from the
donor pool for model fitting for that variable.
If TRUE, all variables with missing data are imputed sequentially across iterations.
Maximum number of iterations (if sequential is TRUE).
Convergence threshold on the per-variable relative change between
iterations: for numeric variables the mean squared change of the imputed
values divided by the variance of the observed values, for factors the
share of imputed cells whose category changed. The sequential process
stops early once the largest per-variable change stays below eps
for two consecutive iterations. The full iterations-by-variables change
matrix is returned as attr(result, "convergence").
If TRUE, additional columns indicating imputed values (VAR_imp) are added.
If TRUE (default), the full input is returned: columns excluded via
considered_variables are passed through unchanged (original column
order, with any VAR_imp indicators appended), matching kNN(),
hotdeck() and irmi(). Set FALSE to return only the considered
columns (plus their indicators), dropping the rest.
If TRUE, all predicted values across all iterations are stored.
Hyperparameter tuning flag. Can be:
TRUE/FALSE globally
or a list specifying tuning per variable, e.g. list(var1 = TRUE)
Tuning runs once per variable, early in the iteration sequence. With
m > 1, tuning runs once in the first imputation and the chosen
parameters are shared by all m imputations (as in mice);
the resulting vimmi object carries the tuning report in its
tuning_log element.
If TRUE additional debugging output is provided
If TRUE, bootstrap resampling is applied before model fitting to account
for model uncertainty. Defaults to TRUE when m > 1 (each
imputation then refits on a bootstrap sample, giving approximately proper
multiple-imputation draws) and to FALSE for single imputation
(m = 1); set explicitly to override. The bootstrap strategy is
controlled by robustboot.
Most effective with method = "robust". Default: FALSE
Bootstrap strategy when boot = TRUE. Options:
"standard" (classical bootstrap),
"stratified" (good/bad residual split, default),
"residual" (inverse residual weighting).
Imputation uncertainty method applied to numeric predictions:
"pmm" (default since 7.3.0: predictive mean matching, Little 1988
-- a random draw among the 5 donors whose predicted values are
nearest the missing cell's prediction, so imputed values are observed
values; for a no-bootstrap ranger fit the donors are scored by their
out-of-bag predictions),
"none" (deterministic point prediction; the pre-7.3.0 default),
"normalerror" (add N(0, sigma_hat)),
"resid" (add sampled residual),
"midastouch" (covariate-distance-weighted PMM, Siddique & Belin 2008).
Factor targets draw from the predicted class probabilities whenever
uncert != "none" (and are imputed by the most probable class
otherwise). If pmm = TRUE is set, it takes precedence over
uncert.
Variables imputed by method = "restricted" always use
uncert = "none": a value-level draw on top of the constrained
solution would break the validation rules the solver just enforced. The
default is overridden silently, an explicitly requested draw mechanism
warns.
Number of multiple imputations. Default: 1 (single imputation).
When m > 1, returns a vimmi object storing the original
data and imputed values efficiently. Use vim_complete to
extract completed datasets.
Optional single number for reproducibility. Applied once via
set.seed at the start of the call (as in mice), so the
whole run -- including all m imputations -- is reproducible while the
m imputations still differ from each other. Default NULL
leaves the random-number stream untouched.
Optional named list mapping variable names to learner parameter lists
(e.g. list(Sleep = list(num.trees = 300L))). The parameters are
applied to the variable's learner without running the tuner -- use this to
reuse tuning results across calls (each entry of a previous run's
tuning_log carries its chosen parameters in $params).
Used internally by m > 1 to share the first imputation's tuned
parameters across all imputations.
NULL (default) or a vimpute_tune_control object
controlling the tuning of tune = TRUE: evaluation budget, CV folds
of the tuning resampling, tuner and batch size. NULL keeps the
built-in data-size heuristics. With m > 1 it applies to the single
tuning run whose parameters all imputations share. The tuning_log
records the budget and folds used per variable.
Optional per-variable predictor control, the equivalent of mice's
predictorMatrix -- and unlike formula it works for EVERY
method, including ranger and xgboost. Either a named list
mapping a target variable to the character vector of its predictors
(e.g. list(Sleep = c("Dream", "Span"))), or a 0/1 (or logical)
matrix with targets in rows and predictors in columns (compatible with
mice::make.predictorMatrix). Variables without an entry use all
other considered variables. A formula supplied for a variable takes
precedence over its predictors entry (as in mice).
Order in which the variables with missings are imputed:
"asis" (default; column order), "increasing.na" (fewest
missings first), "decreasing.na", or a character vector giving an
explicit permutation of the NA-variables.
NULL or a named list of vimpute_spec objects -- one
per variable, e.g. spec = list(Sleep = vs_ranger(num.trees = 300,
tune = TRUE), NonD = vs_robust(donorcond = ">= 0"), .default =
vs_ranger()). Each spec bundles the variable's method, learner
parameters (validated eagerly), formula/predictors,
tune, PMM settings, makeNA and donorcond; the
reserved name ".default" covers unlisted variables. Compiles to
the flat per-variable arguments, which therefore cannot be given in the
same call.
Eileen Vattheuer, Matthias Templ, Alexander Kowarik
Missingness assumptions. Like all conditional (fully
conditional specification) imputation, vimpute() assumes the data
are MAR (missing at random: the probability of missingness may
depend on observed values) -- which includes MCAR
(missing completely at random) as a special case. Under MNAR
(missingness depending on the unobserved values themselves) imputations
and downstream estimates can be biased, and no imputation method can fix
this from the observed data alone; sensitivity analyses are advisable.
makeMissing generates MCAR/MAR/MNAR missingness in complete
data for exactly such simulation-based checks, and
overimpute diagnoses the calibration of the imputation
model on the observed cells.
Other imputation methods:
hotdeck(),
impPCA(),
imputeCellEM(),
imputeCellIRMI(),
imputeCellM(),
imputeCellMCD(),
imputeCellwise(),
imputeRobust(),
imputeRobustChain(),
irmi(),
kNN(),
matchImpute(),
medianSamp(),
rangerImpute(),
regressionImp(),
sampleCat(),
vimmi,
xgboostImpute()
if (FALSE) {
# Single imputation (default)
x <- vimpute(data = sleep, sequential = FALSE)
# Sequential imputation with 3 iterations
y <- vimpute(data = sleep, sequential = TRUE, nseq = 3)
# Impute only selected variables
z <- vimpute(data = sleep, considered_variables =
c("Sleep", "Dream", "Span", "BodyWgt"), sequential = FALSE)
# Multiple imputation (m = 5) with bootstrap and residual uncertainty
# Returns a vimmi object
result <- vimpute(data = sleep, method = "ranger", sequential = FALSE,
imp_var = FALSE, m = 5, boot = TRUE, uncert = "resid")
print(result)
# Extract completed datasets
d1 <- vim_complete(result, 1) # first imputed dataset
all_d <- vim_complete(result, "all") # list of 5 datasets
long_d <- vim_complete(result, "long") # long format with .imp column
# Fit a model on each imputed dataset
fits <- with(result, lm(Sleep ~ Dream + Span))
# Multiple imputation with robust method and residual uncertainty
result2 <- vimpute(data = sleep, method = "robust", m = 5,
boot = TRUE, robustboot = "stratified",
uncert = "normalerror")
}
Run the code above in your browser using DataLab