Decompose a learner's error into the part that comes from being systematically wrong and the part that comes from being unstable, by refitting it over many resamples of the training set and watching what its predictions do at a fixed test set.
bias_variance(
x,
hyperparameters,
dat_test = NULL,
true_values = NULL,
resampler_config = NULL,
test_p = 0.3,
execution_config = setup_FutureExecution(),
verbosity = 1L
)BiasVariance object.
tabular data: Full dataset, outcome in the last column. Resamples are drawn from it after the test set is held out.
Hyperparameters object: Must be fixed, not a search
space -- this measures one model, not a tuning run.
Optional tabular data: Test set. NULL holds one out of x.
Optional Numeric vector: The true function at the test
cases, in their order. Regression: f(x). Classification: the probability of
the positive class. NULL measures bias against the observed outcome, which
includes irreducible noise.
Optional ResamplerConfig object: How training sets
are drawn. NULL uses 100 stratified bootstraps.
Numeric (0, 1): Fraction of x held out as the test set.
Ignored when dat_test is given.
ExecutionConfig object: Supplies the seed and the
workers the resamples are fitted over.
Integer: If > 0, print messages.
Squared bias is measured against true_values when supplied, and against
the observed outcome otherwise:
true_valuesbias_squared is the bias, exactly. Only
simulated data has true values, so a simulation study measures how a
learner divides its error.
The observed outcome carries the irreducible noise, so what is reported is \(bias^2 + \sigma^2\) and no amount of resampling separates them. It stays comparable between learners on the same data -- they share the same \(\sigma^2\) -- so a ranking is still meaningful even though the level is inflated.
variance is the sample variance across resamples (dividing by
n_resamples - 1), which is unbiased for the variance of the fitted
function. bias_squared is the plug-in estimate, and is inflated by roughly
variance / n_resamples because the mean prediction is itself estimated --
so prefer many resamples over few, and treat a bias_squared of the same
order as variance / n_resamples as indistinguishable from zero.
The decomposition above is for squared error, and 0-1 loss does not
decompose additively into bias and variance. For a binary outcome this
function therefore decomposes the squared error of the predicted
probability of the positive class, which does. true_values is then the
true probability, not the class. Multiclass is not supported.
EDG
For squared-error loss, the expected error at a point decomposes as
$$E[(y - \hat{f}(x))^2] = \sigma^2 + (E[\hat{f}(x)] - f(x))^2 + Var[\hat{f}(x)]$$
-- irreducible noise, squared bias, and variance. This function estimates the
second and third per test case, by fitting the model on n_resamples
resamples of the training set and reading the spread and the average of the
predictions each makes at the same test cases.
The test set is held out once and every resample is drawn from what is left, which is what makes the predictions comparable across resamples: a test set that moved would confound the model's variability with the sample's.
Geman S, Bienenstock E, Doursat R (1992). Neural Networks and the Bias/Variance Dilemma. Neural Computation, 4(1), 1-58. tools:::Rd_expr_doi("10.1162/neco.1992.4.1.1")
set.seed(2026)
n <- 200L
features <- data.frame(a = rnorm(n), b = rnorm(n))
dat <- data.frame(features, y = 2 * features[["a"]] + rnorm(n))
bv <- bias_variance(
dat,
hyperparameters = setup_CART(),
resampler_config = setup_StratBoot(n_resamples = 10L),
verbosity = 0L
)
bv
Run the code above in your browser using DataLab