ModelMap (version 2.3.4)

model.importance.plot: Compares the variable importance of two models with a back to back barchart.

Description

Takes two models and produces a back to back bar chart to compare the importance of the predictor variables. Models can be any combination of Random Forest or Stochastic Gradient Boosting, as long as both models have the same predictor variables.

Usage

model.importance.plot(model.obj.1 = NULL, model.obj.2 = NULL, 
model.name.1 = "Model 1", model.name.2 = "Model 2", imp.type.1 = NULL, 
imp.type.2 = NULL, type.label=TRUE, class.1 = NULL, class.2 = NULL, 
scale.by = "sum", sort.by = "model.obj.1", predList = NULL, 
folder = NULL, PLOTfn = NULL, device.type = NULL, jpeg.res = 72, 
device.width = 7, device.height = 7,cex=par()$cex,...)

Arguments

model.obj.1

R model object. The model object to use for left side of barchart. The model object must be of type RF or SGB and have the same predictors as model.obj.2.

model.obj.2

R model object. The model object to use for right side of barchart. The model object must be of type RF or SGB and have the same predictors as model.obj.1.

model.name.1

String. Label for left side of barchart.

model.name.2

String. Label for right side of barchart.

class.1

String. For binary and categorical random forest models. If the name a class is specified, the class-specific relative influence is used for plot. If class.1 = NULL overall relative influence used for plot.

class.2

String. For binary and categorical random forest models. If the name a class is specified, the class-specific relative influence is used for plot. If class.2 = NULL overall relative influence used for plot.

imp.type.1

Number. Type of importance to use for model 1. Importance type 1 is permutation based, as described in Breiman (2001). Importance type 2 is model based. For RF models is the decrease in node impurities attributable to each predictor variable. For SGB mod

imp.type.2

Number. Type of importance to use for model 2. Importance type 1 is permutation based, as described in Breiman (2001). Importance type 2 is model based. For RF models is the decrease in node impurities attributable to each predictor variable. For SGB mod

type.label

Logical. Should axis labels include importance type for each side of plot.

scale.by

String. Scale by: "max" or "sum". When scale.by="max" the importance are scaled for each model so that the maximum importance for each model fills the graph. When scale.by="sum", the importance for each

sort.by

String. Sort by: "model.obj.1", "model.obj.2", "predList". Gives the order to draw the bars for the predictor variables. When sort.by="model.obj.1" the predictors are sorted largest to smallest based on

predList

String. A character vector of the predictor short names used to build the models. If sort.by="predList", then predList is used to specify the order to draw the predictors in the barchart.

folder

String. The folder used for all output. Do not add ending slash to path string. If folder = NULL (default), a GUI interface prompts user to browse to a folder. To use the working directory, specify folder = getwd().

PLOTfn

String. The file name to use to save the generated graphical plots. If PLOTfn = NULL a default name is generated by pasting model.name.1_model.name.2. The filename can be the full path, or it can be the simple basename, in which

device.type

String or vector of strings. Model validation. One or more device types for graphical output from model validation diagnostics. Current choices: lllll{ "default" default graphics device "jpeg"

jpeg.res

Integer. Pixels per inch for jpeg plots. The default is 72dpi, good for on screen viewing. For printing, suggested setting is 300dpi.

device.width

Integer. The device width for diagnostic plots in inches.

device.height

Integer. The device height for diagnostic plots in inches.

cex

Integer. The cex for plots.

...

Arguments to be passed to methods, such as graphical parameters (see par).

Details

The importance measures used in this plot depend on the model type (RF verses SGB) and the response type (continuous, categorical, or binary). Importance type 1 is permutation based, as described in Breiman (2001). Importance is calculated by randomly permuting each predictor variable and computing the associated reduction in predictive performance using Out Of Bag error for RF models and training error for SGB models. Note that for SGB models permutation based importance measures are still considered experimental. Importance type 2 is model based. For RF models, importance type 2 is calculated by the decrease in node impurities attributable to each predictor variable. For SGB models, importance type 2 is the reduction attributable to each variable in predicting the gradient on each iteration as described in described in Friedman (2001). For SGB models: lllclll{ response type type Importance Measure "continuous" 1 permutation reduction predictive performance "binary" 1 permutation reduction predictive performance "continuous" 2 gradient of loss function reduction of squared error "binary" 2 gradient of loss function reduction in sum of squared error } For RF models: lllclll{ response type type Importance Measure "continuous" 1 permutation %IncMSE "binary" 1 permutation Mean Decrease Accuracy "categorical" 1 permutation Mean Decrease Accuracy "continuous" 2 node impurity Residual sum of squares "binary" 2 node impurity Mean Decrease Gini "categorical" 2 node impurity Mean Decrease Gini } For Random Forest models, if imp.type not specified, importance type defaults to imp.type of 1 - permutation importance. For SGB models, permutation importance is considered experimental so importance defaults to imp.type of 2 - reduction of gradient of the loss function. Also, for binary and categorical Random Forest models, class specific importance plots can be generated by the use of the class argument. Note that class specific importance is only available for Random Forest models with importance type 1.

References

Breiman, L. (2001) Random Forests. Machine Learning, 45:5-32. Friedman, J.H. (2001). Greedy function approximation: a gradient boosting machine. Ann. Stat., 29(5):1189-1232.

See Also

model.build

Examples

###########################################################################
############################# Run this set up code: #######################
###########################################################################

# set seed:
seed=38

# Define training and test files:

qdata.trainfn = system.file("external", "helpexamples","DATATRAIN.csv", package = "ModelMap")

# Define folder for all output:
folder=getwd()	


#identifier for individual training and test data points

unique.rowname="ID"

##################################################################
########## Continuous Response, Continuous Predictors ############
##################################################################

#file names:
MODELfn.RF="RF_Bio_TC"				
MODELfn.SGB="SGB_Bio_TC"

#predictors:
predList=c("TCB","TCG","TCW")	

#define which predictors are categorical:
predFactor=FALSE	

# Response name and type:
response.name="BIO"
response.type="continuous"

########## Build Models #################################

model.obj.RF = model.build( model.type="RF",
                       qdata.trainfn=qdata.trainfn,
                       folder=folder,		
                       unique.rowname=unique.rowname,		
                       MODELfn=MODELfn.RF,
                       predList=predList,
                       predFactor=predFactor,
                       response.name=response.name,
                       response.type=response.type,
                       seed=seed
)

model.obj.SGB = model.build( model.type="SGB",
                       qdata.trainfn=qdata.trainfn,
                       folder=folder,		
                       unique.rowname=unique.rowname,		
                       MODELfn=MODELfn.SGB,
                       predList=predList,
                       predFactor=predFactor,
                       response.name=response.name,
                       response.type=response.type,
                       seed=seed+1
)

############## Make Imortance Plot - RF vs. SGB ###################

model.importance.plot(	model.obj.1=model.obj.RF, 
			model.obj.2=model.obj.SGB, 
			model.name.1="RF Model", 
			model.name.2="SGB Model", 
			scale.by="sum",
			sort.by="predList", 
			predList=predList,
			main="RF verses SGB",
			device.type="default")

########## Make Imortance Plot - RF Importance type 1 vs 2 #######

model.importance.plot(	model.obj.1=model.obj.RF, 
			model.obj.2=model.obj.RF, 
			model.name.1="PercentIncMSE", 
			model.name.2="IncNodePurity",
			imp.type.1=1,
			imp.type.2=2,
			scale.by="sum",
			sort.by="predList", 
			predList=predList,
			main="Imp type 1 vs Imp type 2",
			device.type="default")


##################################################################
########## Categorical Response, Continuous Predictors ###########
##################################################################

#file name:
MODELfn="RF_NLCD_TC"				

#predictors:
predList=c("TCB","TCG","TCW")	

#define which predictors are categorical:
predFactor=FALSE	

# Response name and type:
response.name="NLCD"
response.type="categorical"

########## Build Model #################################

model.obj.NLCD = model.build( model.type="RF",
                       qdata.trainfn=qdata.trainfn,
                       folder=folder,		
                       unique.rowname=unique.rowname,	
                       MODELfn=MODELfn,
                       predList=predList,
                       predFactor=predFactor,
                       response.name=response.name,
                       response.type=response.type,
                       seed=seed)

############## Make Imortance Plot ###################

model.importance.plot(	model.obj.1=model.obj.NLCD, 
			model.obj.2=model.obj.NLCD, 
			model.name.1="NLCD=41", 
			model.name.2="NLCD=42",
			class.1="41",
			class.2="42",
			scale.by="sum",
			sort.by="predList", 
			predList=predList,
			main="Class 41 vs. Class 42",
			device.type="default")