check_new_rater: Check new rater

Description

Function for estimating the reliability of codings for a new rater based on Iota 2

Usage

check_new_rater(
  true_values,
  assigned_values,
  con_step_size = 1e-04,
  con_random_starts = 5,
  con_max_iterations = 5000,
  con_rel_convergence = 1e-12,
  con_trace = FALSE,
  fast = TRUE,
  free_aem = FALSE
)

Value

Returns a list with the following three components: The first component estimates_categorical_level comprises all elements that describe the ratings on a categorical level. The elements are sub-divided into raw estimates and chance-corrected estimates.

raw_estimates

alpha_reliability:: A vector containing the Alpha Reliabilities for each category. These values represent probabilities.

beta_reliability:

A vector containing the Beta Reliabilities for each category. These values represent probabilities.

assignment_error_matrix:

An Assignment Error Matrix containing the conditional probabilities for assigning a unit of category i to categories 1 to n.

iota:

A vector containing the Iota values for each category.

elements_chance_corrected

alpha_reliability:: A vector containing the chance-corrected Alpha Reliabilities for each category.

beta_reliability:

A vector containing the chance-corrected Beta Reliabilities for each category.

The second component estimates_scale_level contains elements to describe the quality of the ratings on a scale level. It contains the following elements:

iota_index:: The Iota Index representing the reliability on a scale level.
iota_index_d4:: The Static Iota Index, which is a transformation of the original Iota Index, in order to consider the uncertainty of estimation.
iota_index_dyn2:: The Dynamic Iota Index, which is a transformation of the original Iota Index, in order to consider the uncertainty of estimation.

The third component information contains important information regarding the parameter estimation. It comprises the following elements:

log_likelihood:: Log-likelihood of the best solution.
convergence:: If estimation converged 0, otherwise 1.
est_true_cat_sizes:: Estimated categorical sizes. This is the estimated amount of the categories.
conformity:: 0 if the solution is in line with assumptions of weak superiority. A number greater 0 indicates the number of violations of the assumption of weak superiority.
random_starts:: Numer of random starts for the EM algorithm.
boundaries:: False if the best solution does not contain boundary values. True if the best solution does contain boundary values
p_boundaries:: Percentage of solutions with boundary values during estimation.
call:: Name of the function that created the object.
n_rater:: Number of raters.
n_cunits:: Number of coding units.

Arguments

true_values: Vector containing the true categories of the coding units. Vector must have the same length as assigned_values.
assigned_values: Vector containing the assigned categories of the coding units. Missing values are currently not supported and have to be omitted from the vector. Vector must have the same length as true_values.
con_step_size: Double for specifying the size for increasing or decreasing the probabilities during the conditioning stage of estimation. This value should not be less than 1e-3.
con_random_starts: Integer for the number of random starts within the condition stage.
con_max_iterations: Integer for the maximum number of iterations during the conditioning stage.
con_rel_convergence: Double for determining the convergence criterion during the conditioning stage. The algorithm stops if the relative change is smaller than this criterion.
con_trace: TRUE for printing progress information on the console during estimations in the conditioning stage. FALSE if you do not want to have this information printed.
fast: Bool If TRUE a fast estimation is applied during the condition stage. This option ignores all parameters beginning with "con_". If FALSE the estimation described in Berding and Pargmann (2022) is used. Default is TRUE.
free_aem: Bool If TRUE the Assignment Error Matrix is estimated in a way ensuring conformity with the assumption of weak superiority. if FALSE the Assignment Error Matrix is freely estimated. TRUE is default.

References

Florian Berding and Julia Pargmann (2022). Iota Reliability Concept of the Second Generation. Measures for Content Analysis Done by Humans or Artificial Intelligences. Berlin:Logos. https://doi.org/10.30819/5581