EM algorithm with latent contamination indicators that jointly estimates clean distribution parameters and identifies cellwise outliers. Each continuous cell has a posterior probability of being clean vs contaminated. The clean distribution is modeled as multivariate normal for continuous variables, with categorical variables handled via conditional multinomial logistic regression.
imputeCellEM(
data,
maxit_em = 100,
eps_em = 0.005,
gamma_init = 3,
eps_init = 0.1,
uncert = "conditional",
conditioning = "weighted",
trust_min = 0.5,
trace = FALSE
)A list with components:
the imputed data.frame
n x p matrix of posterior clean probabilities. Continuous observed cells have values in \([0, 1]\); missing cells and categorical columns have weight 1.
estimated clean location vector (continuous variables only)
estimated clean covariance matrix (continuous variables only)
named numeric vector of estimated per-variable contamination rates (continuous variables only)
logical indicating convergence
number of EM iterations performed
numeric vector of composite (pseudo) log-likelihood values, one per iteration (computed after each M-step). This is a sum of per-variable conditional mixture log-likelihoods, not the proper observed-data joint log-likelihood.
data.frame with missing values (mixed continuous + categorical).
maximum EM iterations (default: 100).
convergence tolerance on the relative change in estimated parameters (mu, Sigma). Default: 5e-3.
initial scale inflation factor for the contamination
distribution. Contaminated cells are modeled as having variance
(gamma * sigma)^2 with gamma > 1. Default: 3.
initial contamination probability per variable (default: 0.1). Must be in \((0, 0.5)\).
imputation uncertainty method: "conditional"
(default) draws from the conditional normal distribution, or
"pmm" uses predictive mean matching.
how observed cells enter the E-step conditional
moments. "weighted" (default) multiplies each conditioning
deviation \(x_{ik} - \mu_k\) by its current cell weight
\(w_{ik}\) -- the posterior-expected clean deviation, so likely
contaminated cells are shrunk towards the mean before they enter
the conditional. "trust" conditions only on trusted cells:
deviations with \(w_{ik} < \) trust_min are set to zero
(hard threshold). "unweighted" uses the raw deviations
(all observed cells fully trusted, as in early versions).
trust threshold in \((0, 1)\) used when
conditioning = "trust" (default: 0.5). Ignored otherwise.
logical; if TRUE, print progress information.
Matthias Templ
The algorithm proceeds as follows:
Initialization. Missing values are filled by
initialise (medians for continuous, modes for
categorical). Initial location and scale are estimated robustly
(median and MAD). Cell weights are initialized to 1.
E-step (for each continuous variable \(j\), each observation \(i\)):
Compute the conditional mean and variance of \(x_{ij}\) given the other continuous variables, using the current \(\mu\) and \(\Sigma\).
For observed cells: compute the posterior probability that the cell is clean vs contaminated, yielding cell weight \(w_{ij}\).
For missing cells: impute from the conditional distribution (with optional PMM).
M-step:
Update \(\mu\): cell-weighted mean.
Update \(\Sigma\): pairwise cell-weighted covariance using \(w_{ij} \cdot w_{ik}\) (not row-level min).
Update contamination rates: \(\varepsilon_j = 1 - \mathrm{mean}(w_{ij})\) over observed cells.
Update contamination scale \(\gamma_j\) from weighted variance of contaminated cells.
For categorical variables: fit weighted multinomial logistic with row weights derived from continuous cell weights.
Convergence. Check relative change in
estimated parameters (\(\mu\), \(\Sigma\)); stop when
below eps_em or after maxit_em iterations.
The observed-data log-likelihood (sum of per-variable
conditional mixture log-likelihoods) is tracked for
diagnostics.
This method differs from cellGMM (Zaccaria et al., 2025) in using a single clean component rather than a mixture of clean clusters, and in supporting mixed continuous + categorical data.
The pseudo_loglik component is a composite (pseudo)
log-likelihood: the sum of per-variable conditional mixture
log-likelihoods, not the proper observed-data joint log-likelihood.
It is useful for monitoring convergence but should not be compared
across models or used for model selection criteria such as AIC/BIC.
This implementation is an ECM (Expectation Conditional Maximization) variant rather than a pure EM algorithm, because the conditional variance in the E-step is computed using the updated Sigma from the current M-step rather than the Sigma from the previous iteration. As a result, strict log-likelihood monotonicity is not guaranteed, but is observed empirically in practice.
A.P. Dempster, N.M. Laird, D.B. Rubin (1977) Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society: Series B, 39(1), 1--38.
C.F.J. Wu (1983) On the Convergence Properties of the EM Algorithm. The Annals of Statistics, 11(1), 95--103.
J. Raymaekers, P.J. Rousseeuw (2024) The cellwise minimum covariance determinant estimator. Journal of the American Statistical Association, 119(548), 2610--2621.
G. Zaccaria, L.A. Garcia-Escudero, F. Greselin, A. Mayo-Iscar (2025) Cellwise outlier detection in heterogeneous populations. Technometrics, 67(4), 643--654.
M. Templ, A. Kowarik, P. Filzmoser (2011) Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, Vol. 55, pp. 2793--2806.
imputeCellIRMI, imputeCellM,
initialise, irmi
Other imputation methods:
hotdeck(),
impPCA(),
imputeCellIRMI(),
imputeCellM(),
imputeCellMCD(),
imputeCellwise(),
imputeRobust(),
imputeRobustChain(),
irmi(),
kNN(),
matchImpute(),
medianSamp(),
rangerImpute(),
regressionImp(),
sampleCat(),
vimmi,
vimpute(),
xgboostImpute()
# \donttest{
data(sleep, package = "VIM")
result <- imputeCellEM(sleep)
head(result$data_imputed)
# Inspect estimated contamination rates
result$epsilon
# Cell weight matrix (1 = clean, low = likely contaminated)
image(result$cellweights, main = "Cell weights")
# Log-likelihood trace
plot(result$pseudo_loglik, type = "b", xlab = "Iteration",
ylab = "Pseudo log-likelihood")
# With predictive mean matching for imputation
result2 <- imputeCellEM(sleep, uncert = "pmm", trace = TRUE)
# Mixed data example
data(testdata)
result3 <- imputeCellEM(testdata$wna)
# }
Run the code above in your browser using DataLab