raters (version 1.0)

concordance: Inter rater agreement among a set of raters

Description

Computes a statistic as an index of inter rater agreement among a set of raters. This procedure is based on a statistic not affected by Kappa paradoxes. It is also possible to evaluate if the agreement is nil using the test argument. The p value can be approximated using the Normal, Chi squared distribution or using Monte Carlo algorithm. Normal approximation and Monte Carlo procedure can be calculated even though the number of observers is not the same for each evaluated subject. Fleiss Kappa is also shown and its confidence interval is available when the number of observes is the same for each classified subject.

Usage

concordance(db, test = "Default", B = 1000, alpha = 0.05)

Arguments

db

n*c matrix or data frame, n subjects c categories. The numbers inside the matrix or data frame indicate how many raters chose a specific category for a given subject. A sum of row indicates the total number of raters who evaluated a given subject.

test

Statistical test to evaluate if the raters make random assignment regardless of the characteristic of each subject. Under null hypothesis, it corresponds to a high percentage of assignment errors. Thus, the expected agreement is weak. Normal approximat

B

Number of iterations for Monte Carlo test.

alpha

Level of significance.

Value

• A list containing the following components:
• $Fleiss
  Fleiss Kappa index.
• $S
  Computed index of inter rater agreement not affected by Kappa paradoxes.
• $Lower
  Lower limit of Fleiss Kappa confidence interval. It is computed only when the number of raters is the same for all subjects.
• $Upper
  Upper limit of Fleiss Kappa confidence interval. It is computed only when the number of raters is the same for all subjects.
• $pvalue
  P value for the statistical test.

References

Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 378-382 Falotico, R. Quatto, P. (2010). On avoiding paradoxes in assessing inter-rater agreement. Italian Journal of Applied Statistics 22, 151-160

Examples

data(diagnostic)
concordance(diagnostic,test="Chisq")
concordance(diagnostic,test="Normal")
concordance(diagnostic,test="MC",B=100)