AutoScore: An Interpretable Machine Learning-Based Automatic Clinical Score Generator

AutoScore is a novel machine learning framework to automate the development of interpretable clinical scoring models. AutoScore consists of six modules: 1) variable ranking with machine learning, 2) variable transformation, 3) score derivation, 4) model selection, 5) domain knowledge-based score fine-tuning, and 6) performance evaluation. The original AutoScore structure is elaborated in this article and its flowchart is shown in the following figure. AutoScore was originally designed for binary outcomes and later extended to survival outcomes and ordinal outcomes. AutoScore could seamlessly generate risk scores using a parsimonious set of variables for different types of clinical outcomes, which can be easily implemented and validated in clinical practice. Moreover, it enables users to build transparent and interpretable clinical scores quickly in a straightforward manner.

Please visit our bookdown page for a full tutorial on AutoScore usage.

Usage

The five pipeline functions constitute the 5-step AutoScore-based process for generating point-based clinical scores for binary, survival and ordinal outcomes.

This 5-step process gives users the flexibility of customization (e.g., determining the final list of variables according to the parsimony plot, and fine-tuning the cutoffs in variable transformation):

• STEP(i): AutoScore_rank()or AutoScore_rank_Survival() or AutoScore_rank_Ordinal() - Rank variables with machine learning (AutoScore Module 1)
• STEP(ii): AutoScore_parsimony() or AutoScore_parsimony_Survival() or AutoScore_parsimony_Ordinal() - Select the best model with parsimony plot (AutoScore Modules 2+3+4)
• STEP(iii): AutoScore_weighting() or AutoScore_weighting_Survival() or AutoScore_weighting_Ordinal() - Generate the initial score with the final list of variables (Re-run AutoScore Modules 2+3)
• STEP(iv): AutoScore_fine_tuning() or AutoScore_fine_tuning_Survival() or AutoScore_fine_tuning_Ordinal() - Fine-tune the score by revising cut_vec with domain knowledge (AutoScore Module 5)
• STEP(v): AutoScore_testing() or AutoScore_testing_Survival() or AutoScore_testing_Ordinal() - Evaluate the final score with ROC analysis (AutoScore Module 6)

We also include several optional functions in the package, which could help with data analysis and result reporting.

Citation

Core paper

• Xie F, Chakraborty B, Ong MEH, Goldstein BA, Liu N. AutoScore: A machine learning-based automatic clinical score generator and its application to mortality prediction using electronic health records. JMIR Medical Informatics 2020; 8(10): e21798.
• Xie F, Ning Y, Liu M, Li S, Saffari SE, Yuan H, Volovici V, Ting DSW, Goldstein BA, Ong MEH, Vaughan R, Chakraborty B, Liu N. A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes. STAR Protocols 2023 Jun; 4(2): 102302.

Method extension

• Xie F, Ning Y, Yuan H, Goldstein BA, Ong MEH, Liu N, Chakraborty B. AutoScore-Survival: Developing interpretable machine learning-based time-to-event scores with right-censored survival data. Journal of Biomedical Informatics 2022; 125: 103959.

• Saffari SE, Ning Y, Xie F, Chakraborty B, Volovici V, Vaughan R, Ong MEH, Liu N, AutoScore-Ordinal: An interpretable machine learning framework for generating scoring models for ordinal outcomes, BMC Medical Research Methodology 2022; 22: 286.

• Ning Y, Li S, Ong ME, Xie F, Chakraborty B, Ting DS, Liu N. A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study. PLOS Digital Health 2022; 1(6): e0000062.

Clinical application

This page provides a collection of clinical applications using AutoScore and its extensions. The application list is categorized according to medical specialties and is updated regularly. However, due to the manual process of updating, we are unable to keep track of all publications.

Contact

• Feng Xie (Email: xief@u.duke.nus.edu)
• Yilin Ning (Email: yilin.ning@duke-nus.edu.sg)
• Nan Liu (Email: liu.nan@duke-nus.edu.sg)

Package installation

Install from GitHub or CRAN:

# From Github
install.packages("devtools")
library(devtools)
install_github(repo = "nliulab/AutoScore", build_vignettes = TRUE)

# From CRAN (recommended)
install.packages("AutoScore")

Load AutoScore package:

library(AutoScore)