ezECM (version 1.0.0)

becm_decision: B-ECM performance metrics

Description

Outputs batch performance metrics of decisions using the output of predict.BayesECM() when the true category of testing events are known. Can be used for empirically comparing different model fits.

Usage

becm_decision(
  bayes_pred = NULL,
  vic = NULL,
  cat_truth = NULL,
  alphatilde = 0.05,
  pn = TRUE,
  C = matrix(c(0, 1, 1, 0), ncol = 2),
  rej = NULL
)

Value

A list of two data frames of logicals. The rows in each data frame correspond to the rows in bayes_pred$Ytilde. The first data frame, named results, has three columns named correct, fn, and fp. The results column indicates if the categorization is correct. fn and fp stand for false negatives and false positives respectively. fn and fp are found under binary categorization. Values of NA are returned when a false positive or false negative is not relevant. The second data frame, named rej, indicates the rejection of each new observation from each category via typicality index. One can use rej to inspect if the typicality index played a role in categorization, or to supply to another call of becm_decision.

Arguments

bayes_pred

An object of class "BayesECMpred" returned from the predict.BayesECM() function. Data and order of data provided to predict.BayesECM() must match the order of the cat_truth argument.

vic

Character string, indicating the "Very Important Category" (vic) used for calculating categorization metrics. The return of becm_decision() provides information on if new observations should be categorized into vic or the remainder of the training categories grouped together.

cat_truth

Vector of the same length as nrow(bayes_pred$Ytilde), where Ytilde was previously supplied to the predict.BayesECM() function. Used for accuracy calculations of a BayesECM fit and decision criteria when the true category of the new observations are known.

alphatilde

Numeric scalar between 0 and 1 used for the significance level of typicality indices.

pn

Logical. Acronym for "false Positive, false Negative". pn == FALSE indicates that only accuracy of categorizations should be returned, while pn == TRUE indicates that false positives and false negatives should be returned in addition to accuracy.

C

Square matrix of dimension 2 or the number of categories. Used as the loss function in the decision theoretic framework. The default is 0-1 loss for binary categorization.

rej

data.frame of rejection via typicality index retrieved from a previous call to becm_decision(). Useful for saving computation time when comparing the results from different supplied matrices for argument C, while using a constant value of alphatilde.

Details

The matrix C specifies the loss for a set of categorization actions for a single new observation \(\tilde{y}_{\tilde{p}}\) given the probability of \(\tilde{y}_{\tilde{p}}\) belonging to each of \(K\) categories. Actions are specified as the columns, and the event category random variables are specified as the rows. See the vignette vignette("syn-data-code", package = "ezECM") for more mathematical details.

The dimension of matrix C specifies the categorization type. A dimension of 2 is binary categorization, with the first column and row always corresponding to the category chosen as the vic argument. Otherwise, when the dimension of C is equal to the number of categories, the indices of the rows and columns of C are in the same order as the categories listed for names(bayes_pred$BayesECMfit$Y).

Examples

csv_use <- "good_training.csv"
file_path <- system.file("extdata", csv_use, package = "ezECM")
training_data <- import_pvals(file = file_path, header = TRUE, sep = ",", training = TRUE)

trained_model <- BayesECM(Y = training_data, BT = c(10,1000))

csv_use <- "good_newdata.csv"
file_path <- system.file("extdata", csv_use, package = "ezECM")
new_data <- import_pvals(file = file_path, header = TRUE, sep = ",", training = TRUE)

bayespred <- predict(trained_model,  Ytilde = new_data)

accuracy <- becm_decision(bayes_pred = bayespred, vic = "explosion", cat_truth = new_data$event)