DAAGxtras (version 0.8-2)

compareModels: Compare accuracy of alternative classification methods

Description

Compare, between models, probabilities that the models assign to membership in the correct group or class. Probabilites should be estimated from cross-validation or from bootstrap out-of-bag data or preferably for test data that are completely separate from the data used to dervive the model.

Usage

compareModels(groups = fgl$type, estprobs = list(lda = NULL, rf = NULL),
            gpnames = NULL, robust = TRUE, print = TRUE)

Arguments

groups

Factor that specifies the groups

estprobs

List whose elements (with names that identify the models) are matrices that give for each observation (row) estimated probabilities of membership for each of the groups (columns).

gpnames

Character: names for groups, if different from levels(groups)

robust

Logical, TRUE or FALSE

print

Logical. Should results be printed?

Value

• modelAVS
  Average accuracies for models
• modelSE
  Approximate average SE for comparing models
• gpAVS
  Average accuracies for groups
• gpSE
  Approximate average SE for comparing groups
• obsEff
  Effects assigned to individual observations

Details

The estimated probabilities are compared directly, under normal distribution assumptions. An effect is fitted for each observation, plus an effect for the method. Comparison on a logit scale may sometimes be preferable. An option to allow this is scheduled for incorporation in a later version.

Examples

library(MASS)
library(DAAG)
ldahat <- lda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
qdahat <- qda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
rfhat <- predict(randomForest(species ~ length+breadth, data=cuckoos),
                 type="prob")
compareModels(groups=cuckoos$species, estprobs=list(lda=ldahat,
              qda=qdahat, rf=rfhat), robust=FALSE)