rminer (version 1.1)

mmetric: Compute classification or regression error metrics.

Description

Compute classification or regression error metrics.

Usage

mmetric(y, x = NULL, metric, D = 0.5, TC = -1, val = NULL)

Arguments

y

if there are predictions (!is.null(x)), y should be a numeric vector or factor with the target desired responses (or output values). Else, y should be a list returned by the mini

x

the predictions (should be a numeric vector if task="reg", matrix if task="prob" or factor if task="class" (use if y is not a list).

metric

a R function or if a character valid options are:

• AUC-- overall area under the curve (of ROC curve, classification).
• NAUC-- normalized AUC (given a fixedval=FPR, classification).

D

decision threshold (for task="prob", probabilistic classification) within [0,1]. The class is TRUE if prob>D.

TC

target class (for multi-class classification class) within {1,...,Nc}, where Nc is the number of classes. Notes: if TC==-1 (the default value), then it is assumed:

• ifmetricis "AUC

val

auxiliary value, check the metric argument for details.

Value

• Returns the computed error metric(s).

Details

Compute classification or regression error metrics:

• mmetric-- compute one error metric given y and x or given a mining list.
• metrics-- computes several classification or regression metrics. Additional arguments are:
  • AUC-- if TRUE compute the AUC metrics.
  • BRIER-- if TRUE compute the Brier score.
  • task--seefit.
  Inmetrics,taucis the global AUC andacc classis the each class accuracy. Themetricsusage is:metrics(y, x, D = 0.5, TC = -1, AUC = TRUE, BRIER = FALSE, Run=1, task = "default") where:
  • Run-- is the Run of mining execution ifyis a mining object.
  • ifAUC=TRUEorBRIER=TRUEthen AUC/BRIER statistics are also computed.

References

• To check for more details about rminer and for citation purposes: P. Cortez. Data Mining with Neural Networks and Support Vector Machines Using the R/rminer Tool. In P. Perner (Ed.), Advances in Data Mining - Applications and Theoretical Aspects 10th Industrial Conference on Data Mining (ICDM 2010), Lecture Notes in Artificial Intelligence 6171, pp. 572-583, Berlin, Germany, July, 2010. Springer. ISBN: 978-3-642-14399-1. @Springer:http://www.springerlink.com/content/e7u36014r04h0334 http://www3.dsi.uminho.pt/pcortez/2010-rminer.pdf
• About the Brier and Global AUC scores: A. Silva, P. Cortez, M.F. Santos, L. Gomes and J. Neves. Rating Organ Failure via Adverse Events using Data Mining in the Intensive Care Unit. In Artificial Intelligence in Medicine, Elsevier, 43 (3): 179-193, 2008. http://dx.doi.org/10.1016/j.artmed.2008.03.010
• About the classification and regression metrics: I. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2005.
• About the forecasting metrics: R. Hyndman and A. Koehler Another look at measures of forecast accuracy. In International Journal of Forecasting, 22(4):679-688, 2006.

See Also

fit, predict.fit, mining, mgraph, savemining and Importance.

Examples

### regression
y=c(1:94,95.01,96.1,97.2,97.8,99.3,99.7);x=rnorm(100,5,0.3)+y
print(mmetric(y,x,"MAE"))
print(mmetric(y,x,"RMSE"))
print(mmetric(y,x,"TOLERANCE",val=5))
print(mmetric(y[95:100],x[95:100],"THEILSU2",val=y[1:94])) # b = in-samples
print(mmetric(y[95:100],x[95:100],"MASE",val=y[1:94])) # b = in-samples
print(metrics(y,x)) # several regression metrics
# user defined error function example:
# myerror = number of samples with absolute error above 10% of y: 
myerror=function(y,x){return (sum(abs(y-x)>(0.1*y)))}
print(mmetric(y,x,metric=myerror))

### binary classification 
y=factor(c("a","a","b","b"))
x=factor(c("a","b","b","b"))
print(mmetric(y,x,"CONF"))
print(mmetric(y,x,"ACC"))
print(metrics(y,x))
px=matrix(nrow=4,ncol=2)
px[1,]=c(0.7,0.3)
px[2,]=c(0.4,0.6)
px[3,]=c(0.7,0.3)
px[4,]=c(0.3,0.7)
print(px)
print(mmetric(y,px,"CONF"))
print(mmetric(y,px,"ACC"))
print(mmetric(y,px,"CONF",D=0.7,TC=2))
print(metrics(y,px,D=0.7,TC=2))
px2=px[,2]
print(px2)
print(mmetric(y,px,"CONF"))
print(mmetric(y,px2,"CONF",D=0.7,TC=2))
print(mmetric(y,px,"AUC"))
print(mmetric(y,px2,"AUC"))
print(mmetric(y,px2,"AUC",TC=2))

### multi-class classification 
data(iris)
M=mining(Species~.,iris,model="dt",Runs=2)
print(mmetric(M,metric="ACC",TC=2,D=0.7))
print(mmetric(M,metric="CONF",TC=2,D=0.7))
print(mmetric(M,metric="AUC",TC=3))
print(mmetric(M,metric="AUC",TC=1))
print(mmetric(M,metric="TPR",TC=1))
print(mmetric(M,metric="TPRATFPR",TC=3,val=0.05))
print(mmetric(M,metric="NAUC",TC=3,val=0.05))
print(mmetric(M,metric="ALIFT",TC=3))
print(mmetric(M,metric="ALIFTATPERC",TC=3,val=0.1))
print(mmetric(M,metric="NALIFT",TC=3,val=0.1))
print(metrics(M,BRIER=TRUE,Run=1)) # several Run 1 classification metrics
print(metrics(M,BRIER=TRUE,Run=1,TC=3)) # several Run 1 TC=3 classification metrics