Learn R Programming

CompositeReliability

Determine the Composite Reliability of a Naturalistic, Unbalanced Dataset

The reliability of assessment tools is a crucial aspect of monitoring student performance in various educational settings. It ensures that the assessment outcomes accurately reflect a student's true level of performance. However, when assessments are combined, determining composite reliability can be challenging, especially for naturalistic and unbalanced datasets. This package is designed to estimate composite reliability using multivariate generalizability theory and enhance the analysis of assessment data. The package allows for the inclusion of weight per assessment type and produces extensive G- and D-study results with graphical interpretations, and options to find the set of weights that minimize the standard error of measurement (SEM).

library("CompositeReliability")

G-Study

The "GStudy" function, available in the "CompositeReliability" package, is designed to assess the reliability coefficient and the standard error of measurement (SEM) for each assessment type. This function utilizes the harmonic mean of the number of assessments per type as a measure of effective assessment quantity. By providing a dataset as input, users can obtain reliable estimates of the reliability coefficient and SEM for each assessment type. The desired number of decimal places in the output (nrDigitsOutput) can be added as input. NB: The dataset "mydata" is included in the package as example.

GStudy(mydata,nrDigitsOutput=4,optimizeSEM=TRUE)

The output presents a table with descriptive statistics for each Type included in the dataset.

To view the observed variances, covariances, and error scores, the dataset, along with a vector specifying the number of assessments per assessment type, can be added as input.

varcov <- calculateVarCov(mydata, n=c("A"=3.1579, "B"=3.0000, "C"=1.4286))
varcov$S_p
varcov$S_iINp
varcov$S_delta

D-Study

The number of assessments per Type has a notable impact on the reliability estimation. To investigate the relationship between the number of assessments and reliability, the D-study functionality within the program provides an analysis of the reliability coefficient and standard error of measurement (SEM) for varying numbers of assessments per type.

plots <- DStudy(mydata, maxNrAssessments = 60)
plots$plotRel
plots$plotSEM

Composite reliability

The estimation of composite reliability, encompassing the integration of student results from assessments of various types, is performed utilizing multivariate generalizability theory. Leveraging the capabilities of the R package, researchers can obtain a comprehensive analysis that includes the determination of weights optimizing the standard error of measurement (SEM) and the subsequent presentation of the composite reliability value. NB: make sure that the sum of weights included in vector 'w' is equal to 1.

compRel <- computeCompositeReliability(mydata, n=c("A"=10, "B"=5, "C"=2), w=c("A"=1/3,"B"=1/3, "C"=1/3), optimizeSEM=TRUE)
compRel$reliability
compRel$SEM
compRel$weights

And it is possible to compute the maximum value of the composite reliability coefficient of the dataset, given the number of assessments per type.

compMaxRel <- computeMaxCompositeReliability(mydata, n=c("A"=10, "B"=5, "C"=2))
compMaxRel$reliability
compMaxRel$SEM
compMaxRel$weights

Copy Link

Version

Install

install.packages('CompositeReliability')

Monthly Downloads

165

Version

1.0.3

License

GPL (>= 3)

Issues

0

Pull Requests

0

Stars

0

Forks

0

Maintainer

Joyce Moonen - van Loon

Last Published

August 21st, 2023

Functions in CompositeReliability (1.0.3)

DStudy

DStudy: the program presents the reliability coefficient and the SEM for different numbers of assessments per type. Both the reliability coefficient and the SEM are presented in graphs for differing numbers of assessments, given insight in the impact on the reliability if more or less assessments per type were required or advised.
GStudy

GStudy for a dataset in which every student p has a potentially differing number of scores i on each assessment type m. i.e. model i: (p x m). The output gives descriptive statistics, reliability coefficient and SEM for each assessment type.
computeCompositeReliability

computeCompositeReliability: multivariate generalizability theory approach to estimate the composite reliability of student performance across different types of assessments.
mydata

mydata
computeMaxCompositeReliability

computeMaxCompositeReliability: multivariate generalizability theory approach to estimate the maximum composite reliability of student performance across different types of assessments.
calculateVarCov

calculateVarCov: Estimate variance and covariance components of assessee p S_p and mean assessment scores i nested in assessees S_iINp, and determine the error scores S_delta
checkDatasets

checkDatasets: assert that the given datasets adhere to the assumptions and requirements of this package i.e. the data set 'mydata' is a dataframe with 3 columns, named "ID", "Type" and "Score", column "Score" contains numeric data, and each combination of "ID" and "Type" exists at least once, data set n contains a numerical value for each "Type", and data set weights contains a numerical value for each "Type" and the sum of all values is equal to 1.
GStudyPerType

GStudyPerType: This function is mainly used within calculateVarCov.R, but can be executed on its own to determine the reliability coefficient and SEM for a dataset with a single type of assessment.
calculateReliability

calculateReliability: determine the reliability and SEM per Type
%>%

Pipe operator