Greedy variable selection for ridge regression
Variable Selection for Ridge Regression using Forward Greedy, Backward Greedy, and Adaptive Forward-Backward Greedy (FoBa) Methods
foba(x,y, type=c("foba","foba.aggressive", "foba.conservative", "forward","backward"), steps=0, intercept=TRUE, nu=0.5,lambda=1e-10)
- matrix of predictors
- One of "foba", "foba.aggressive", "foba.conservative", "forward", or "backward". The names can be abbreviated to any unique substring. Default is "foba".
- Number of greedy (forward+backward) steps. Default is the number of variables for forward and backward, and twice the number of variables for foba.
- If TRUE, an intercept is included in the model (and not penalized), otherwise no intercept is included. Default is TRUE.
- In range (0,1): controls how likely to take a backward step (more likely when nu is larger). Default is 0.5.
- Regularization parameter for ridge regression. Default is 1e-5.
FoBa for least squares regression is described in [Tong Zhang (2008)]. This implementation supports ridge regression. The "foba" method takes a backward step when the ridge penalized risk increase is less than nu times the ridge penalized risk reduction in the corresponding backward step. The "foba.conservative" method takes a backward step when the risk increase is less than nu times the smallest risk reduction in all previous forward steps. The "foba.aggressive" method takes a backward step when the cumulative risk changes in backward step is less than nu times the changes in the forward steps.
A "foba" object is returned, which contains the following components:
- The function call resulting to the object
- Which variable selection method is used
- The variable selection path: a sequence of variable addition/deletions
- Coefficients (ridge regression solution) at each step with selected features
- Zero if intercept=FALSE, and the mean of x if intercept=TRUE
- Zero if intercept=FALSE, and the mean of y if intercept=TRUE
Tong Zhang (2008) "Adaptive Forward-Backward Greedy Algorithm for Learning Sparse Representations", Rutgers Technical Report (long version).
Tong Zhang (2008) "Adaptive Forward-Backward Greedy Algorithm for Sparse Learning with Linear Models", NIPS'08 (short version).
print.foba and predict.foba methods for foba
data(boston) model.foba <- foba(boston$x,boston$y,steps=20) print(model.foba) model.foba.a <- foba(boston$x,boston$y,type="foba.a",steps=20) # Can use abbreviations print(model.foba.a) model.for <- foba(boston$x,boston$y,type="for",steps=20) print(model.for) model.back <- foba(boston$x,boston$y,type="back") # Use only first 20 variables print(model.back)