Setup hyperparameters for LINAD, the Linear Additive Tree. LINAD generalizes the decision tree and the regularized linear model: CART, the Additive Tree, and a penalized linear fit are each a special case of it, recovered exactly by the hyperparameter combinations given under Details.
setup_LINAD(
max_leaves = 20L,
min_cases_split = 2L,
min_cases_leaf = 1L,
min_cases_node_model = NULL,
nvmax = NULL,
forward_stop = NULL,
lambda = NULL,
alpha = NULL,
learning_rate = 0.1,
root_nvmax = NULL,
root_lambda = NULL,
root_alpha = NULL,
root_learning_rate = 1,
n_cuts = NULL,
gamma = 0.1,
split_binning = NULL,
split_bin_type = "frequency",
node_test = NULL,
split_criterion = NULL,
line_search = "expansion",
constant_rule = "closed_form",
ifw = FALSE,
patience = NULL,
force_max_leaves = FALSE,
smooth_validation_curve = NULL,
node_model = "forward",
root_model = NULL,
split_search = "stump",
split_features = NULL,
linear_features = NULL,
global_features = NULL,
line_search_max = 1000,
root_forward_stop = NULL,
root_node_test = NULL
)LINADHyperparameters object.
(Tunable) Integer [1, Inf): Largest number of terminal nodes to grow.
(Tunable) Integer [2, Inf): Fewest cases a node may hold and still be considered for splitting.
(Tunable) Integer [1, Inf): Fewest cases a split must leave on each side.
(Tunable) Optional Integer [1, Inf): Fewest cases needed to fit a linear model at a node. Applies only when node_model is not "constant". NULL uses 10.
(Tunable) Optional Integer [1, Inf): Most terms forward selection may add beside the intercept. Applies only when node_model is "forward". NULL uses 3.
(Tunable) Optional Character {"bic", "aic", "none"}: Criterion a term must improve to be added by forward selection. Applies only when node_model is "forward". NULL uses "bic".
(Tunable) Optional Numeric [0, Inf): L2 penalty on the leaf models. Applies only when node_model is "forward", "ridge" or "elasticnet". NULL uses 0.05.
(Tunable) Optional Numeric [0, 1]: Elastic-net mixing, 0 ridge to 1 lasso. Applies only when node_model is "elasticnet". NULL uses 1.
(Tunable) Numeric (0, 1]: Shrinkage applied to every functional update.
(Tunable) Optional Integer [1, Inf): nvmax for the root model. NULL uses nvmax.
(Tunable) Optional Numeric [0, Inf): lambda for the root model. NULL uses lambda.
(Tunable) Optional Numeric [0, 1]: alpha for the root model. NULL uses alpha.
(Tunable) Numeric [0, 1]: Shrinkage applied to the root model's slopes.
(Tunable) Optional Integer [2, Inf): Cut points tried per feature; split_bin_type decides their spacing. Applies only when split_search is "exhaustive". NULL uses 20.
(Tunable) Numeric [0, 1]: Weight a case retains in the branch it does not belong to. 0 is a hard partition.
(Tunable) Optional Integer [2, Inf): Discretize each numeric feature into this many bins and consider only bin boundaries as splits. Applies to both split searches.
(Tunable) Character {"frequency", "width"}: How bin edges are placed.
(Tunable) Optional Character {"none", "aic", "bic"}: Information criterion a node's slopes must improve over the constant alone. Applies only when node_model is "ridge" or "elasticnet"; forward selection uses forward_stop. NULL uses "none".
(Tunable) Optional Character {"mean", "linear"}: What the stump search scores a candidate side by. Applies only when split_search is "stump". NULL uses "mean".
(Tunable) Character {"expansion", "child", "none"}: Scope of the Newton step for each update.
(Tunable) Character {"closed_form", "least_squares"}: How a node's constant is computed. The two coincide for regression and differ only for classification.
(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.
(Tunable) Optional Integer [1, Inf): Expansions without validation improvement before growth stops. Requires a validation set. NULL disables patience-based early stopping.
Logical: If TRUE, keep every leaf grown instead of selecting a size on the validation set.
Optional Logical: If TRUE, smooth the validation curve before reading its minimum. Applies only when force_max_leaves is FALSE. NULL does not smooth.
Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at each node. "constant" is the intercept-only model every decision tree node carries; the others add a regularized linear model on top of it.
Optional Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at the root. NULL uses node_model.
Character {"stump", "exhaustive"}: How a split is chosen.
Optional Character vector: Features that may define a split. NULL imposes no constraint, so every feature may.
Optional Character vector: Features that get a slope in the node models. NULL imposes no constraint, so every feature does.
Optional Character vector: Linear features whose slope is shared by every leaf. Must be a subset of linear_features where that is set. NULL imposes no constraint, so no coefficient is pinned.
Numeric (0, Inf): Largest absolute step the line search may take.
(Tunable) Optional Character {"bic", "aic", "none"}: forward_stop for the root model. NULL uses forward_stop, or "bic" where that is NULL. Applies only when the root model is "forward".
(Tunable) Optional Character {"none", "aic", "bic"}: node_test for the root model. NULL uses node_test, or "none" where that is NULL. Applies only when the root model is "ridge" or "elasticnet".
Each of the following is a special case of LINAD, recovered exactly:
max_leaves = 1. Nothing is split, so the model
is the root's alone -- ridge, elastic net or forward stepwise selection,
whichever node_model names. Verified against a direct penalized solve.
node_model = "constant", gamma = 0, learning_rate = 1,
line_search = "none". Constants only, a hard partition and no
shrinkage. Fitted values match rpart's to machine precision.
The same, with gamma > 0. Soft weighting is
precisely what separates the Additive Tree from CART (Luna and
colleagues, 2019; see References).
Any node_model other than "constant", with
max_leaves > 1 -- a tree whose nodes each carry a linear model.
Every setting named above is tunable, so the data can select where a dataset
belongs in that space:
a linear generating process should select a full root fit and no splitting,
while terminal nodes needing opposite-signed coefficients should select a
split first. root_learning_rate is the dial for that first step -- at 0 the
root is the outcome's mean and the tree splits first, at 1 a full linear
model is fitted before any split.
LINAD fits a regularized linear model on the whole sample, splits on the gradient of the loss, fits a model on each side of the split, and repeats. Every update is linear, so the coefficients along a root-to-leaf path sum: the finished model is a tree with one linear model per leaf, optimized stagewise rather than fitted leaf by leaf. Prediction routes a case to its leaf and evaluates that leaf's coefficients.
Splits are found on the features as given, so a factor splits on a set of its levels; the node models use a reference-coded design matrix built internally. LINAD therefore needs no encoding preprocessor and handles factors directly.
max_leaves plays the role that the number of trees plays in gradient
boosting, and like it is chosen on held-out data: the tree is grown to
max_leaves and the size with the lowest validation loss is kept, unless
force_max_leaves is TRUE. Pass dat_validation to train to control the
set that choice is made on.
gamma lets every case carry weight into both branches, decaying as
gamma^depth, so each node's model is pulled toward the fit to the rest of
the data, which reduces variance.
split_search = "exhaustive" scores each candidate split by the loss after
fitting both child models, rather than taking the best split of the gradient.
It costs more per split and detects an interaction that changes another
feature's slope without changing either side's mean, which leaves no trace
in any one feature's marginal fit.
A node's model is fitted to what its parent left, so its slopes are the
change to the parent's coefficients and lambda and alpha penalize that
change. At alpha = 1 the change is a lasso, so a feature whose coefficient
does not need to move gets exactly zero and inherits its parent's value, so
each feature's effect adapts or stays fixed per node. Ridge
shrinks every change but zeroes none, so every coefficient moves at every
node.
node_test selects, node by node, between a linear model and a constant:
the slopes are kept only where they improve the node's information
criterion, and the node carries a constant otherwise. The resulting tree
carries coefficients only in regions where the response is locally linear.
The test can reduce accuracy; the default is "none".
It applies to the shrinking models only. Forward selection applies the same
criterion per term through forward_stop, and a set of terms that each
passes it also passes it jointly.
split_criterion sets what the stump scores a side by. "mean" scores the
level its fit explains, the CART criterion, which does not detect curvature:
a parabola's two halves have the same mean, so the split at the vertex
scores near last. "linear" adds the slope in the split variable, at one
further pass per feature per node, and on such a problem recovers the
exhaustive search's fit. It does not detect the cross-variable case above,
and where the change belongs to another feature it can select a split on
the wrong one. The default is "mean".
split_binning discretizes numeric features before either search, so a
candidate split falls on a bin boundary rather than between any two distinct
values; n_cuts then thins what is left, for the exhaustive search only.
split_bin_type governs both: "frequency" spaces cuts evenly through the
cases, "width" evenly through the feature's range.
A feature's behavior has two independent axes: whether it may define a partition, and whether its modeled effect is absent, shared by every subgroup, or free to change along a path. Three name-based selectors set them, and on each of the three NULL imposes no constraint, which means "all" for the first two and "nothing pinned" for the third:
split_featuresFeatures the tree may split on.
linear_featuresFeatures that receive a slope in the node models.
global_featuresLinear features whose slope is shared by every
leaf: the root estimates it and no node below may change it. Must be a
subset of linear_features where that is set.
The axes are independent, so a feature may split without entering any node model, enter every node model without ever splitting, do both, or neither. Roles are assigned to source features: naming a factor moves all of its encoded columns together, never one of them.
Declaring effect scope turns LINAD into a varying-coefficient model with a
hierarchical form: a global baseline plus the node deviations accumulated
down the path. global is a hard constraint on the coefficient and holds
exactly at any gamma. gamma is a soft constraint, pulling every node's
fit toward the fit to the rest of the data until, at gamma = 1, every
coefficient is effectively global. The two are independent: gamma sets how
far a node's adaptive coefficients may move, and global_features sets
which coefficients may move.
Because a global coefficient is estimated at the root, root_learning_rate
shrinks it: at 0 the root fits no slopes and a global feature carries no
effect anywhere.
The constrained family has these reductions. No splits (max_leaves = 1)
is a penalized linear model; node_model = "constant" is a tree of
constants; making every linear feature global gives a tree with shared
coefficients and node-specific constants; and leaving all three NULL
reproduces unconstrained LINAD exactly.
The split search cost is linear in the number of candidate features, so restricting the split set reduces training time on wide data.
learning_rate shrinks the update made at each node. It does not shrink
the root, which root_learning_rate governs and which defaults to 1: the
root model is the initialization F_0, not a boosting step.
Consequently, for any learning_rate, the fit includes the full global
linear model at the root. On data whose signal is largely linear, shrinking
the tree's correction changes the metrics little, and different small
learning rates give similar fits. On a signal the root cannot represent,
such as a step with no linear component, the same sweep moves test
R-squared from 0.58 to 0.97.
To shrink everything, including the root, set root_learning_rate to the
same value. A small rate then under-fits: at 1e-6 the fit reduces to the
constant that minimizes the loss (the weighted mean of the outcome, or half
its log odds for a classification), which scores an R-squared of 0. root_learning_rate interpolates between that constant and
the full linear model, never towards zero.
The line_search step does not counteract this. It is scale-free (for
squared error it is the exact minimizer along the update direction, near 1
for any residual size), so the applied step stays proportional to
learning_rate. Measured node updates scale linearly with it, by a factor
of 50 across a 50-fold change in the rate.
line_search and constant_rule each select among
alternative update rules -- the scope of the Newton step, which frontier node
is split next, and how a node's constant is computed. Each is tunable.
EDG
Every node carries a constant, as any decision tree's nodes do. node_model
selects the model fitted at each node: "constant" for that constant alone,
or a regularized linear model on top of it. With max_leaves and gamma,
these settings span a hyperparameter space that contains the following
algorithms and the models between them.
Luna JM, Gennatas ED, Ungar LH, Eaton E, Diffenderfer ES, Jensen ST, Simone CB 2nd, Friedman JH, Solberg TD, Valdes G (2019). Building more accurate decision trees with the additive tree. Proceedings of the National Academy of Sciences, 116(40), 19887-19893. tools:::Rd_expr_doi("10.1073/pnas.1816748116")
linad_hyperparams <- setup_LINAD(max_leaves = 10L, learning_rate = 0.1)
linad_hyperparams
# The Additive Tree is LINAD without the linear models
addtree_hyperparams <- setup_LINAD(node_model = "constant", gamma = 0.8)
addtree_hyperparams
Run the code above in your browser using DataLab