sirt (version 0.47-36)

rm.hrm: Hierachical Rater Model Based on Signal Detection Theory (HRM-SDT)

Description

This function estimates a version of the hierarchical rater model (HRM) based on signal detection theory (HRM-SDT; DeCarlo, 2005; DeCarlo, Kim & Johnson, 2011).

Usage

rm.hrm(dat, pid, rater, Qmatrix = NULL, theta.k = seq(-9, 9, len = 30), 
    est.a.item = FALSE, est.c.rater = "n", est.d.rater = "n", 
    d.min = 0.5, d.max = 100, d.start = 3, max.increment = 1, numdiff.parm = 0.00001, 
    maxdevchange = 0.1, globconv = .001, maxiter = 1000, msteps = 4, mstepconv = 0.001)

## S3 method for class 'rm.hrm':
summary(object,...)

Arguments

dat

Original data frame. Ratings on variables must be in rows, i.e. every row corresponds to a person-rater combination.

pid

Person identifier.

rater

Rater identifier.

Qmatrix

An optional Q-matrix. If this matrix is not provided, then by default the ordinary scoring of categories (from 0 to the maximum score of $K$) is used.

theta.k

A grid of theta values for the ability distribution.

est.a.item

Should item parameters $a_i$ be estimated?

est.c.rater

Type of estimation for item-rater parameters $c_{ir}$ in the signal detection model. Options are 'n' (no estimation), 'e' (set all parameters equal to each other), 'i' (item wise estmation), 'r' (rat

est.d.rater

Type of estimation of $d$ parameters. Options are the same as in est.c.rater.

d.min

Minimal $d$ parameter to be estimated

d.max

Maximal $d$ parameter to be estimated

d.start

Starting value of $d$ parameters

max.increment

Maximum increment of item parameters during estimation

numdiff.parm

Numerical differentiation step width

maxdevchange

Maximum relative deviance change as a convergence criterion

globconv

Maximum parameter change

maxiter

Maximum number of iterations

msteps

Maximum number of iterations during an M step

mstepconv

Convergence criterion in an M step

object

Object of class rm.hrm

...

Further arguments to be passed

Value

• A list with following entries:
• deviance
  Deviance
• ic
  Information criteria and number of parameters
• item
  Data frame with item parameters. The columns N and M denote the number of oberved ratings and the observed mean of all ratings, respectively. In addition to item parameters $\tau_{ik}$ and $a_i$, the mean for the latent response (latM) is computed as $E( \eta_i ) = \sum_p P( \theta_p ) q_{ik} P( \eta_i = k | \theta_p )$ which provides an item parameter at the original metric of ratings. The latent standard deviation (latSD) is computed in the same manner.
• rater
  Data frame with rater parameters. Transformed $c$ parameters (c_x.trans) are computed as $c_{irk} / ( d_{ir} \cdot K )$.
• person
  Data frame with person parameters: EAP and corresponding standard errors
• EAP.rel
  EAP reliability
• sigma
  Standard deviation of the trait
• tau.item
  Item parameters $\tau_{ik}$
• se.tau.item
  Standard error of item parameters $\tau_{ik}$
• a.item
  Item slopes $a_i$
• se.a.item
  Standard error of item slopes $a_i$
• c.rater
  Rater parameters $c_{irk}$
• se.c.rater
  Standard error of rater severity parameter $c_{irk}$
• d.rater
  Rater slope parameter $d_{ir}$
• se.d.rater
  Standard error of rater slope parameter $d_{ir}$
• f.yi.qk
  Individual likelihood
• f.qk.yi
  Individual posterior distribution
• probs
  Item probabilities at grid theta.k. Note that these probabilities are calculated on the pseudo items $i \times r$, i.e. the interaction of item and rater.
• prob.item
  Probabilities $P( \eta_i = \eta | \theta )$ of latent item responses evaluated at theta grid $\theta_p$.
• n.ik
  Expected counts
• pi.k
  Estimated trait distribution $P(\theta_p)$.
• maxK
  Maximum number of categories
• procdata
  Processed data
• iter
  Number of iterations
• ...
  Further values

Details

The specification of the model follows DeCarlo et al. (2011). The second level models the ideal rating (latent response) $\eta =0, ...,K$ of person $p$ on item $i$ $$P( \eta_{pi} = \eta | \theta_p ) \propto exp( a_{i} q_{ik} \theta_p - \tau_{ik} )$$ At the first level, the ratings $X_{pir}$ for person $p$ on item $i$ and rater $r$ are modelled as a signal detection model $$P( X_{pir} \le k | \eta_{pi} ) = G( c_{irk} - d_{ir} \eta_{pi} )$$ where $G$ is the logistic distribution function and the categories are $k=1,\ldots , K+1$. The thresholds $c_{irk}$ can be further restricted to $c_{irk} = c_{k}$ (est.c.rater='e'), $c_{irk} = c_{ik}$ (est.c.rater='i') or $c_{irk} = c_{ir}$ (est.c.rater='r'). The same holds for rater precision parameters $d_{ir}$.

References

DeCarlo, L. T. (2005). A model of rater behavior in essay grading based on signal detection theory. Journal of Educational Measurement, 42, 53-76. DeCarlo, L. T. (2010). Studies of a latent-class signal-detection model for constructed response scoring II: Incomplete and hierarchical designs. ETS Research Report ETS RR-10-08. Princeton NJ: ETS. DeCarlo, T., Kim, Y., & Johnson, M. S. (2011). A hierarchical rater model for constructed responses, with a signal detection rater model. Journal of Educational Measurement, 48, 333-356.

See Also

The facets rater model can be estimated with rm.facets.

Examples

#############################################################################
# EXAMPLE 1: Hierarchical rater model (HRM-SDT) data.ratings1
#############################################################################
data(data.ratings1)
dat <- data.ratings1

# Model 1: Partial Credit Model: no rater effects
mod1 <- rm.hrm( dat[ , paste0( "k",1:5) ] , rater=dat$rater , 
            pid=dat$idstud , est.c.rater="n" , est.d.rater="n" , 
            d.start=100 , maxiter=15)
summary(mod1)
            
# Model 2: Generalized Partial Credit Model: no rater effects
mod2 <- rm.hrm( dat[ , paste0( "k",1:5) ] , rater=dat$rater , 
            pid=dat$idstud , est.c.rater="n" , est.d.rater="n" , 
            est.a.item =TRUE , d.start=100 , maxiter=15)
summary(mod2)
            
# Model 3: Equal effects in SDT
mod3 <- rm.hrm( dat[ , paste0( "k",1:5) ] , rater=dat$rater , 
            pid=dat$idstud , est.c.rater="e" , est.d.rater="e" , maxiter=15)
summary(mod3)
  ##   Item Parameters 
  ##      item   N     M tau.Cat1 tau.Cat2 tau.Cat3 a  latM latSD
  ##   k1   k1 274 1.573   -2.164   -2.275    0.304 1 1.476 1.085
  ##   k2   k2 274 1.336   -1.883   -0.865    2.638 1 1.249 1.022
  ##   k3   k3 274 1.529   -4.414   -3.754   -0.274 1 1.481 0.872
  ##   k4   k4 274 1.372   -1.519   -1.366    3.544 1 1.234 0.974
  ##   k5   k5 274 1.412   -2.506   -2.213    2.986 1 1.300 0.894
  ##   -----------------------------------------------------------------
  ##   Rater Parameters 
  ##           item.rater  N     M     d   c_1   c_2   c_3 c_1.trans c_2.trans c_3.trans
  ##   k1-db01    k1-db01 41 0.854 3.593 0.645 5.475 9.723      0.06     0.508     0.902
  ##   k2-db01    k2-db01 41 0.951 3.593 0.645 5.475 9.723      0.06     0.508     0.902
  ##   k3-db01    k3-db01 41 1.122 3.593 0.645 5.475 9.723      0.06     0.508     0.902
  ##   k4-db01    k4-db01 41 1.024 3.593 0.645 5.475 9.723      0.06     0.508     0.902
  ##   k5-db01    k5-db01 41 1.146 3.593 0.645 5.475 9.723      0.06     0.508     0.902
  ##   k1-db02    k1-db02 37 1.351 3.593 0.645 5.475 9.723      0.06     0.508     0.902

# Model 4: Rater effects in SDT
mod4 <- rm.hrm( dat[ , paste0( "k",1:5) ] , rater=dat$rater , 
            pid=dat$idstud , est.c.rater="r" , est.d.rater="r" , maxiter=15)
summary(mod4)