SEMdeep (version 1.0.0)

crossValidation: Cross-validation of linear SEM, ML or DNN training models

Description

The function does a R-repeated K-fold cross-validation of SEMrun(), SEMml() or SEMdnn() models.

Usage

crossValidation(
  models,
  outcome = NULL,
  K = 5,
  R = 1,
  metric = NULL,
  ncores = 2,
  verbose = FALSE,
  ...
)

Value

A list of 2 objects: (1) "stats", a list with performance evaluation metrics. If outcome=FALSE, mean and (0.025;0.0975)-quantiles of amse, r2, and srmr across folds and repetitions are reported; if outcome=TRUE, mean and (0.025;0.0975)-quantiles of f1, accuracy and mcc from confusion matrix averaged across all repetitions are reported; and (2) "PE", a data.frame of repeated cross-validation results.

Arguments

models

A named list of model fitting objects from SEMrun(), SEMml() or SEMdnn() function, with default group=NULL (for SEMrun() or outcome=NULL (for SEMml() or SEMdnn()).

outcome

A character vector (as.factor) of labels for a categorical output (target). If NULL (default), the categorical output (target) will not be considered.

K

A numerical value indicating the number of k-fold to create.

R

A numerical value indicating the number of repetitions for the k-fold cross-validation.

metric

A character value indicating the metric for boxplots display, i.e.: "amse", "r2", or "srmr", for continuous outcomes, and "f1", "accuracy" or "mcc", for a categorical outcome (default = NULL).

ncores

Number of cpu cores (default = 2).

verbose

Output to console boxplots and summarized results (default = FALSE).

...

Currently ignored.

Author

Mario Grassi

Details

Easy-to-use model comparison and selection of SEM, ML or DNN models, in which several models are defined and compared in a R-repeated K-fold cross-validation procedure. The winner model is selected by reporting the mean predicted performances across all runs, as outline in de Rooij & Weeda (2020).

References

de Rooij M, Weeda W. Cross-Validation: A Method Every Psychologist Should Know. Advances in Methods and Practices in Psychological Science. 2020;3(2):248-263. doi:10.1177/2515245919898466

Examples

# \donttest{
# Load Amyotrophic Lateral Sclerosis (ALS)
ig<- alsData$graph
data<- alsData$exprs
data<- transformData(data)$data
group<- alsData$group

# ... with continuous outcomes 

res1 <- SEMml(ig, data, algo="tree")
res2 <- SEMml(ig, data, algo="rf")
res3 <- SEMml(ig, data, algo="xgb")
res4 <- SEMml(ig, data, algo="nn")

models <- list(res1,res2,res3,res4)
names(models) <- c("tree","rf","xgb","nn")

res.cv1 <- crossValidation(models, outcome=NULL, K=5, R=10)
print(res.cv1$stats)

#... with a categorical (as.factor) outcome

outcome <- factor(ifelse(group == 0, "control", "case"))
res.cv2 <- crossValidation(models, outcome=outcome, K=5, R=10)
print(res.cv2$stats)
# }