VSURF (version 0.6)

VSURF: Variable Selection Using Random Forests

Description

Three steps variable selection procedure based on random forests for supervised classification and regression problems. First step ("thresholding step") is dedicated to eliminate irrelevant variables from the dataset. Second step ("interpretation step") aims to select all variables related to the response for interpretation prupose. Third step ("prediction step") refines the selection by eliminating redundancy in the set of variables selected by the second step, for prediction prupose.

Usage

VSURF(x, y, ntree=500,
      mtry=if (!is.factor(y)) max(floor(ncol(x)/3), 1)
           else floor(sqrt(ncol(x))),
      nfor.thres=50, nmin=1, nfor.interp=25, nsd=1, nfor.pred=25, nmj=1)

Arguments

x

A data frame or a matrix of predictors, the columns represent the variables.

y

A response vector (must be a factor for classification problems and numeric for regression ones).

ntree

Number of trees in each forests grown. Standard parameter of randomForest.

mtry

Number of variables randomly sampled as candidates at each split. Standard parameter of randomForest.

nfor.thres

Number of forests grown for "thresholding step" (first of the three steps).

nmin

Number of times the "minimum value" is multiplied to set threshold value.

nfor.interp

Number of forests grown for "intepretation step" (second of the three steps).

nsd

Number of times the standard deviation of the minimum value of err.interp is multiplied.

nfor.pred

Number of forests grown for "prediction step" (last of the three steps).

nmj

Number of times the mean jump is multiplied.

Value

• An object of class VSURF, which is a list with the following components:
• varselect.thres
  A vector of indexes of variables selected after "thresholding step", sorted according to their mean VI, in decreasing order.
• imp.varselect.thres
  A vector of importances of the varselect.thres variables.
• min.thres
  The minimum predicted value of a pruned CART tree fitted to the curve of the standard deviations of VI.
• num.varselect.thres
  Number of variables selected by "thresholding step".
• ord.imp
  A list containing the order of all variables mean importance. $x contains the mean importances sorted in decreasing order. $ix contains indexes of the variables.
• ord.sd
  A vector of standard deviations of all variables importance. The order is given by ord.imp.
• mean.perf
  Mean OOB error rate, obtained by a random forests build on all variables.
• pred.pruned.treee
  Predictions of the CART tree fitted to the curve of the standard deviations of VI.
• varselect.interp
  A vector of indexes of variables selected after "interpretation step".
• err.interp
  A vector of the mean OOB error rates of the embedded random forests models build during the "interpretation step".
• sd.min
  The standard deviation of OOB error rates associated to the random forests model attaining the minimum mean OOB error rate during the "interpretation step".
• num.varselect.interp
  Number of variables selected by "interpretation step".
• varselect.pred
  A vector of indexes of variables selected after "prediction step".
• err.pred
  A vector of the mean OOB error rates of the random forests models build during the "prediction step".
• mean.jump
  The mean jump value computed during the "prediction step".
• num.varselect.pred
  Number of variables selected by "prediction step".
• nmin
  Number of times the "minimum value" is multiplied to set threshold value.
• nsd
  Number of times the standard deviation of the minimum value of err.interp is multiplied.
• nmj
  Number of times the mean jump is multiplied.
• comput.time
  Overall computation time

Details

• First step ("thresholding step"): first,nfor.thresrandom forests are computed using the functionrandomForestwith argumentsimportance=TRUE. Then variables are sorted according to their mean variable importance (VI), in decreasing order. This order is kept all along the procedure. Next, a threshold is computed:min.thres, the minimum predicted value of a pruned CART tree fitted to the curve of the standard deviations of VI. Finally, the actual "thresholding step" is performed: only variables with a mean VI larger thannmin*min.thresare kept.
• Second step ("intepretation step"): the variables selected by the first step are considered.nfor.interpembedded random forests models are grown, starting with the random forest build with only the most important variable and ending with all variables selected in the first step. Then,err.minthe minimum mean out-of-bag (OOB) error of these models and its associated standard deviationsd.minare computed. Finally, the smallest model (and hence its corresponding variables) having a mean OOB error less thanerr.min+nsd*sd.minis selected.
• Third step ("prediction step"): the starting point is the same than in the second step. However, now the variables are added to the model in a stepwise manner.mean.jump, the mean jump value is calculated using variables that have been left out by the second step, and is set as the mean absolute difference between mean OOB errors of one model and its first following model. Hence a variable is included in the model if the mean OOB error decrease is larger thannmj*mean.jump.

References

Genuer, R. and Poggi, J.M. and Tuleau-Malot, C. (2010), Variable selection using random forests, Pattern Recognition Letters 31(14), 2225-2236

See Also

plot.VSURF, summary.VSURF, VSURF.thres, VSURF.interp, VSURF.pred

Examples

data(iris)
iris.vsurf <- VSURF(x=iris[,1:4], y=iris[,5], ntree=100, nfor.thres=20,
                    nfor.interp=10, nfor.pred=10)
iris.vsurf

# A more interesting example with toys data (see ?toys)
# (less than 1 min to execute)
data(toys)
toys.vsurf <- VSURF(x=toys$x, y=toys$y)
toys.vsurf