Data derived from the paper GPT detectors are biased against non-native English writers. The study authors carried out a series of experiments passing a number of essays to different GPT detection models. Juxtaposing detector predictions for papers written by native and non-native English writers, the authors argue that GPT detectors disproportionately classify real writing from non-native English writers as AI-generated.
data_detectors(...)tibble
Arguments passed to pins::pin_read().
data_detectors()
#> # A tibble: 6,185 x 9
#> kind .pred_AI .pred_class detector native name model document_id prompt
#> <fct> <dbl> <fct> <chr> <chr> <chr> <chr> <dbl> <chr>
#> 1 Human 1.00 AI Sapling No Real~ Human 497 <NA>
#> 2 Human 0.828 AI Crossplag No Real~ Human 278 <NA>
#> 3 Human 0.000214 Human Crossplag Yes Real~ Human 294 <NA>
#> 4 AI 0 Human ZeroGPT <NA> Fake~ GPT3 671 Plain
#> 5 AI 0.00178 Human Originality~ <NA> Fake~ GPT4 717 Eleva~
#> 6 Human 0.000178 Human HFOpenAI Yes Real~ Human 855 <NA>
#> 7 AI 0.992 AI HFOpenAI <NA> Fake~ GPT3 533 Plain
#> 8 AI 0.0226 Human Crossplag <NA> Fake~ GPT4 484 Eleva~
#> 9 Human 0 Human ZeroGPT Yes Real~ Human 781 <NA>
#> 10 Human 1.00 AI Sapling No Real~ Human 460 <NA>
#> # i 6,175 more rows
tibble::glimpse(data_detectors())
#> Rows: 6,185
#> Columns: 9
#> $ kind <fct> Human, Human, Human, AI, AI, Human, AI, AI, Human, Human, ~
#> $ .pred_AI <dbl> 9.999942e-01, 8.281448e-01, 2.137465e-04, 0.000000e+00, 1.~
#> $ .pred_class <fct> AI, AI, Human, Human, Human, Human, AI, Human, Human, AI, ~
#> $ detector <chr> "Sapling", "Crossplag", "Crossplag", "ZeroGPT", "Originali~
#> $ native <chr> "No", "No", "Yes", NA, NA, "Yes", NA, NA, "Yes", "No", NA,~
#> $ name <chr> "Real TOEFL", "Real TOEFL", "Real College Essays", "Fake C~
#> $ model <chr> "Human", "Human", "Human", "GPT3", "GPT4", "Human", "GPT3"~
#> $ document_id <dbl> 497, 278, 294, 671, 717, 855, 533, 484, 781, 460, 591, 11,~
#> $ prompt <chr> NA, NA, NA, "Plain", "Elevate using technical", NA, "Plain~
A data frame with 6,185 rows and 9 columns:
Whether the essay was written by a "Human" or "AI".
The class probability from the GPT detector that the inputted text was written by AI.
The uncalibrated class prediction, encoded as
if_else(.pred_AI > .5, "AI", "Human")
The name of the detector used to generate the predictions.
For essays written by humans, whether the essay was written
by a native English writer or not. These categorizations are coarse;
values of "Yes" may actually be written by people who do not write with
English natively. NA indicates that the text was not written by a human.
A label for the experiment that the predictions were generated from.
For essays that were written by AI, the name of the model that generated the essay.
A unique identifier for the supplied essay. Some essays were supplied to multiple detectors. Note that some essays are AI-revised derivatives of others.
For essays that were written by AI, a descriptor for the form of "prompt engineering" passed to the model.
# \donttest{
data_detectors()
# }
Run the code above in your browser using DataLab