The NIH dataset (Erosheva et al., 2020a) was sampled from
a full set of 54,740 R01 applications submitted by black and white
principal investigators (PIs) and reviewed by Center for Scientific
Review (CSR) of the National Institutes of Health (NIH) during council
years 2014--2016.
It contains the original random sample of white applicants as generated by Erosheva et al. (2020b) and a sample of 46 black applicants generated to obtain the same ratio of white and black applicants as in the original sample (for details, see Erosheva et al., 2021a). The dataset was used by Erosheva et al. (2021b) to demonstrate issues of inter-rater reliability in case of restricted samples.
The available variables include preliminary criterion scores on Significance, Investigator, Innovation, Approach, Environment and a preliminary Overall Impact Score. Each of these criteria and the overall score is scored on an integer scale from 1 (best) to 9 (worst). Besides the preliminary criteria and Overall Impact Scores, the data include applicant race, the structural covariates (PI ID, application ID, reviewer ID, administering institute, IRG, and SRG), the matching variables -- gender, ethnicity (Hispanic/Latino or not), career stage, type of academic degree, institution prestige (as reflected by the NIH funding bin), area of science (as reflected by the IRG handling the application), application type (new or renewal) and status (amended or not) -- as well as the final overall score. In addition, the file includes a study group ID variable that refers to the Matched and Random subsets used in the original study.
NIHNIH is a data.frame consisting of 5802 observations on
27 variables.
Proposal ID.
Preliminary Overall Impact score (1-9 integer scale, 1 best).
Preliminary Criterion Scores (1-9 integer scale, 1 best).
Principal investigator's self-identified race; "White"
or "Black".
Anonymized ID of principal investigator (PI).
PI's gender membership; "Male" or "Female".
PI's ethnicity; "Hispanic/Latino" or "Non-Hispanic".
PI's career stage; "ESI" Early Stage Investigator,
"Experienced" Experienced Investigator, or "Non-ES NI"
Non-Early Stage New Investigator.
PI's degree; "PhD", "MD", "MD/PhD",
or "Others".
Lead PI's institution's FY 2014 total institution NIH funding; 5 bins with 1 being most-funded.
Group ID.
Reviewer's ID.
IRG (Integrated Research Group) id.
Administering Organization id.
SRG (Scientific Research Group) id.
Application type, "New" or "Renewal".
Ammend. Logical.
Average of the three overall scores from different reviewers.
Average of the three overall scores from different reviewers, increased by multiple of 0.001 of the worst score.
Project rank calculated based on ScoreAvg.
Project rank calculated based on ScoreAvgAdj.
Final Overall Impact score (1-9 integer scale, 1 best; "ND"
refers to "not discussed")
Final Overall Impact score (1-9 integer scale, 1 best).
Erosheva, E. A., Grant, S., Chen, M.-C., Lindner, M. D., Nakamura, R. K., & Lee, C. J. (2020a). NIH peer review: Criterion scores completely account for racial disparities in overall impact scores. Science Advances 6(23), eaaz4868, tools:::Rd_expr_doi("10.1126/sciadv.aaz4868")
Erosheva, E. A., Grant, S., Chen, M.-C., Lindner, M. D., Nakamura, R. K., & Lee, C. J. (2020b). Supplementary material: NIH peer review: Criterion scores completely account for racial disparities in overall impact scores. Science Advances 6(23), eaaz4868, tools:::Rd_expr_doi("10.17605/OSF.IO/4D6RX")
Erosheva, E., Martinkova, P., & Lee, C. J. (2021a). Supplementary material: When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review.
Erosheva, E., Martinkova, P., & Lee, C. J. (2021b). When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society -- Series A. Accepted.
ICCrestricted()