fleiss_kappa

This function calculates Fleiss' Kappa.

In social and educational settings, the use of Artificial
Intelligence (AI) is a challenging task. Relevant data is often only
available in handwritten forms, or the use of data is restricted by
privacy policies. This often leads to small data sets. Furthermore, in
the educational and social sciences, data is often unbalanced in terms
of frequencies. To support educators as well as educational and social
researchers in using the potentials of AI for their work, this package
provides a unified interface for neural nets in 'PyTorch' to deal with
natural language problems. In addition, the package ships with a shiny
app, providing a graphical user interface. This allows the usage of
AI for people without skills in writing python/R scripts. The tools
integrate existing mathematical and statistical methods for dealing
with small data sets via pseudo-labeling (e.g. Cascante-Bonilla et al.
(2020) <doi:10.48550/arXiv.2001.06001>) and imbalanced data via the
creation of synthetic cases (e.g. Bunkhumpornpat et al. (2012)
<doi:10.1007/s10489-011-0287-y>). Performance evaluation of AI is
connected to measures from content analysis which educational and
social researchers are generally more familiar with (e.g. Berding &
Pargmann (2022) <doi:10.30819/5581>, Gwet (2014)
<ISBN:978-0-9708062-8-4>, Krippendorff (2019)
<doi:10.4135/9781071878781>). Estimation of energy consumption and CO2
emissions during model training is done with the 'python' library
'codecarbon'. Finally, all objects created with this package allow to
share trained AI models with other people.

Berding Florian

aifeducation

Artificial Intelligence for Education

Tykhonova Yuliia

Pargmann Julia

Leube Anna

Riebenbauer Elisabeth

Rebmann Karin

Slopinski Andreas

fleiss_kappa function

<dl><dt>rater_one</dt>
<dd><code>factor</code> rating of the first coder.</dd>
<dt>rater_two</dt>
<dd><code>factor</code> ratings of the second coder.</dd>
<dt>additional_raters</dt>
<dd><code>list</code> Additional raters with same requirements as <code>rater_one</code> and <code>rater_two</code>. If
there are no additional raters set to <code>NULL</code>.</dd></dl>

Arguments

Calculate Fleiss' Kappa — fleiss_kappa

<dl>

<dt>rater_one</dt>
<dd><code>factor</code> rating of the first coder.</dd>


<dt>rater_two</dt>
<dd><code>factor</code> ratings of the second coder.</dd>


<dt>additional_raters</dt>
<dd><code>list</code> Additional raters with same requirements as <code>rater_one</code> and <code>rater_two</code>. If
there are no additional raters set to <code>NULL</code>.</dd>

</dl>

fleiss_kappa: Calculate Fleiss' Kappa

Description

Usage

Value

Arguments

References

See Also