ShinyItemAnalysis (version 1.5.4)

ICCrestricted: Range-restricted reliability with intra-class correlation

Description

Function estimating reliability with intra-class correlation for the complete or for the range-restricted sample.

Usage

ICCrestricted(
  Data,
  case,
  var,
  rank = NULL,
  dir = "top",
  sel = 1,
  nsim = 100,
  ci = 0.95,
  seed = NULL
)

Value

A data.frame with the following columns:

n_sel

number of ratees selected/subsetted.

prop_sel

proportion of ratees selected.

dir

direction of range-restriction. NA if range is effectively not restricted (100% used).

VarID

variance due to ratee, "true variance", between-group variance.

VarResid

residual variance.

VarTotal

total variance.

ICC1

single-rater inter-rater reliability.

ICC1_LCI

lower bound of the confidence interval for ICC1.

ICC1_UCI

upper bound of the confidence interval for ICC1.

ICC3

multiple-rater inter-rater reliability.

ICC3_LCI

lower bound of the confidence interval for ICC3.

ICC3_UCI

upper bound of the confidence interval for ICC3.

Arguments

Data

matrix or data.frame which includes variables describing ID of ratees (specified in case), ratings (specified in var), and (optionally) rank of ratees (specified in rank).

case

character: name of the variable in Data with ID of the ratee (subject or object being evaluated, such as a respondent, proposal, patient, applicant etc.)

var

character: name of the variable in Data with the ratings/scores.

rank

numeric: vector of ranks of ratees. If not provided, rank of ratee is calculated based on average rating based on var variable.

dir

character: direction of range-restriction, available options are "top" (default) or "bottom". Can be an unambiguous abbreviation (i.e., "t" or "b").

sel

numeric: selected number (given > 1) or percentage (given <= 1) of ratees. Default value is 1 (complete dataset).

nsim

numeric: number of simulations for bootstrap confidence interval. Default value is 100.

ci

numeric: confidence interval. Default value is 0.95.

seed

seed for simulations. Default value is NULL, random seed. See lme4::bootMer() for more detail.

Author

Patricia Martinkova
Institute of Computer Science of the Czech Academy of Sciences

Jan Netik
Institute of Computer Science of the Czech Academy of Sciences

References

Erosheva, E., Martinkova, P., & Lee, C. (2021a). When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society - Series A. Accepted.

Erosheva, E., Martinkova, P., & Lee, C. (2021b). Supplementary material for When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review.

Examples

# ICC for the whole sample
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj")

# ICC for the range-restricted sample considering 80% of top ratees
ICCrestricted(
  Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
  sel = 0.8
)