Setup hyperparameters for LINADForest: a bagged ensemble of Linear Additive
Trees. Each tree is fitted on a bootstrap sample of the cases, restricted to
mtry_tree of the features, searching mtry_split of them at each split,
and chooses its own number of leaves on the cases its bootstrap left out.
setup_LINADForest(
n_trees = 50L,
mtry_split = NULL,
mtry_tree = NULL,
max_leaves = 20L,
min_cases_split = 2L,
min_cases_leaf = 1L,
min_cases_node_model = NULL,
nvmax = NULL,
forward_stop = NULL,
lambda = NULL,
alpha = NULL,
learning_rate = 1,
root_nvmax = NULL,
root_lambda = NULL,
root_alpha = NULL,
root_learning_rate = 1,
n_cuts = NULL,
gamma = 0.1,
split_binning = NULL,
split_bin_type = "frequency",
node_test = NULL,
split_criterion = NULL,
line_search = "expansion",
constant_rule = "closed_form",
ifw = FALSE,
patience = NULL,
force_max_leaves = FALSE,
smooth_validation_curve = NULL,
node_model = "forward",
root_model = NULL,
split_search = "stump",
split_features = NULL,
linear_features = NULL,
global_features = NULL,
line_search_max = 1000,
root_forward_stop = NULL,
root_node_test = NULL
)LINADForestHyperparameters object.
(Tunable) Integer [1, Inf): Number of trees in the forest.
(Tunable) Optional Integer [1, Inf): Features sampled at each split search. NULL scans every feature the tree holds.
(Tunable) Optional Integer [1, Inf): Features available to a whole tree. NULL gives every tree every feature.
(Tunable) Integer [1, Inf): Largest number of terminal nodes to grow in each tree.
(Tunable) Integer [2, Inf): Fewest cases a node may hold and still be considered for splitting.
(Tunable) Integer [1, Inf): Fewest cases a split must leave on each side.
(Tunable) Optional Integer [1, Inf): Fewest cases needed to fit a linear model at a node. Applies only when node_model fits one. NULL uses 10.
(Tunable) Optional Integer [1, Inf): Most terms forward selection may add beside the intercept. Applies only when node_model is "forward". NULL uses 3.
(Tunable) Optional Character {"bic", "aic", "none"}: Criterion a term must improve to be added by forward selection. Applies only when node_model is "forward". NULL uses "bic".
(Tunable) Optional Numeric [0, Inf): L2 penalty on the node models. Applies only when node_model is "forward", "ridge" or "elasticnet". NULL uses 0.05.
(Tunable) Optional Numeric [0, 1]: Elastic-net mixing, 0 ridge to 1 lasso. Applies only when node_model is "elasticnet". NULL uses 1.
(Tunable) Numeric (0, 1]: Shrinkage applied to every functional update.
(Tunable) Optional Integer [1, Inf): nvmax for the root model. NULL uses nvmax.
(Tunable) Optional Numeric [0, Inf): lambda for the root model. NULL uses lambda.
(Tunable) Optional Numeric [0, 1]: alpha for the root model. NULL uses alpha.
(Tunable) Numeric [0, 1]: Shrinkage applied to the root model's slopes.
(Tunable) Optional Integer [2, Inf): Cut points tried per feature. Applies only when split_search is "exhaustive". NULL uses 20.
(Tunable) Numeric [0, 1]: Weight a case retains in the branch it does not belong to. 0 is a hard partition.
(Tunable) Optional Integer [2, Inf): Discretize each numeric feature into this many bins and consider only bin boundaries as splits.
(Tunable) Character {"frequency", "width"}: How bin edges are placed.
(Tunable) Optional Character {"none", "aic", "bic"}: Information criterion a node's slopes must improve over the constant alone. Applies only when node_model is "ridge" or "elasticnet"; forward selection uses forward_stop. NULL uses "none".
(Tunable) Optional Character {"mean", "linear"}: What the stump search scores a candidate side by. Applies only when split_search is "stump". NULL uses "mean".
(Tunable) Character {"expansion", "child", "none"}: Scope of the Newton step for each update.
(Tunable) Character {"closed_form", "least_squares"}: How a node's constant is computed.
(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.
(Tunable) Optional Integer [1, Inf): Expansions without validation improvement before growth stops. Requires a validation set. NULL disables patience-based early stopping.
Logical: If TRUE, grow every tree to max_leaves instead of selecting a size on its out-of-bag cases.
Optional Logical: If TRUE, smooth the out-of-bag curve before reading its minimum. Applies only when force_max_leaves is FALSE. NULL does not smooth.
Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at each node.
Optional Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at each tree's root. NULL uses node_model.
Character {"stump", "exhaustive"}: How a split is chosen.
Optional Character vector: Features that may define a split. NULL imposes no constraint, so every feature may. mtry_split samples within this set.
Optional Character vector: Features that get a slope in the node models. NULL imposes no constraint, so every feature does.
Optional Character vector: Linear features whose slope is shared by every leaf of a tree. Must be a subset of linear_features where that is set. NULL imposes no constraint.
Numeric (0, Inf): Largest absolute step the line search may take.
(Tunable) Optional Character {"bic", "aic", "none"}: forward_stop for the root model. NULL uses forward_stop, or "bic" where that is NULL. Applies only when the root model is "forward".
(Tunable) Optional Character {"none", "aic", "bic"}: node_test for the root model. NULL uses node_test, or "none" where that is NULL. Applies only when the root model is "ridge" or "elasticnet".
Three sources of variation, each acting at a different level:
Each tree is fitted on n cases drawn with
replacement, so roughly 37% of cases are out-of-bag for any given tree.
mtry_treeEach tree is restricted to a random subset of the features, for its splits and its node models alike. A tree's coefficients then cover only the features it held.
mtry_splitEach split search sees a fresh sample of the features
the tree holds, which decorrelates the trees further. If that sample
finds no split that reduces the loss, the node is searched again over
every feature, so mtry_split changes which split is made and never
whether a node can split.
Both mtry_* default to NULL, meaning every feature.
A LINAD tree picks its number of leaves on held-out data, as gradient
boosting picks its number of trees. Each tree selects its size on its own
out-of-bag cases, so no validation set is required. force_max_leaves = TRUE turns the selection off and grows every tree to max_leaves.
The same out-of-bag predictions give the fitted forest an error estimate with
no resampling, reported by print().
learning_rate defaults to 1, where a single tree uses 0.1. A single LINAD
tree shrinks each node update to control its variance; in a forest,
averaging controls variance, so each tree is a strong learner, as a random
forest's trees are unpruned.
LINADForest parallelizes across trees, by process rather than by thread.
n_workers_algorithm therefore has an effect only when the run is not
already dispatching: inside a parallel outer fold the forest runs its trees
sequentially, since a fold may not dispatch again. For a threaded backend
such as Ranger the same setting means threads within each fold.
EDG
Every tree-level hyperparameter of setup_LINAD is available here and tunable. The size of each tree is selected per tree from its out-of-bag cases, so the defaults are usable without tuning.
Wager S, Hastie T, Efron B (2014). Confidence intervals for random forests: the jackknife and the infinitesimal jackknife. Journal of Machine Learning Research, 15, 1625-1651.
linadforest_hyperparams <- setup_LINADForest(n_trees = 25L)
linadforest_hyperparams
Run the code above in your browser using DataLab