Lightweight Reproducible Reporting
Order, create and store reports from R. By defining a
lightweight interface around the inputs and outputs of an
analysis, a lot of the repetitive work for reproducible research
can be automated. We define a simple format for organising and
describing work that facilitates collaborative reproducible
research and acknowledges that all analyses are run multiple
times over their lifespans.
- an attendant in a hospital responsible for the non-medical care of patients and the maintenance of order and cleanliness.
- a soldier who carries orders or performs minor tasks for an officer.
orderly is a package designed to help make analysis more reproducible. Its principal aim is to automate a series of basic steps in the process of writing analyses, making it easy to:
- track all inputs into an analysis (packages, code, and data resources)
- store multiple versions of an analysis where it is repeated
- track outputs of an analysis
- create analyses that depend on the outputs of previous analyses
orderly we have two main hopes:
- analysts can write code that will straightforwardly run on someone else's machine (or a remote machine)
- when an analysis that is run several times starts behaving differently it will be easy to see when the outputs started changing, and what inputs started changing at the same time
orderly requires a few conventions around organisation of a project, and after that tries to keep out of your way. However, these requirements are designed to make collaborative development with git easier by minimising conflicts and making backup easier by using an append-only storage system.
One often-touted goal of R over point-and-click analyses packages is that if an analysis is scripted it is more reproducible. However, essentially all analyses depend on external resources - packages, data, code, and R itself; any change in these external resources might change the results. Preventing such changes in external resources is not always possible, but tracking changes should be straightforward - all we need to know is what is being used.
- the source file (
- templates used for styling
- data that is read in for the analysis
- code that is directly read in with
orderly package helps by
- collecting external resources before an analysis
- ensuring that all required external resources are identified
- removing any manual work in tracking information about these external resources
- allowing running reports multiple times and making it easy to see what changed and why
The core problem is that analyses have no general interface. Consider in contrast the role that functions take in programming. All functions have a set of arguments (inputs) and a return value (outputs). With
orderly, we borrow this idea, and each piece of analysis will require that the user describes what is needed and what will be produced.
The user describes the inputs of their analysis, including:
- SQL queries (if using databases)
- Required R sources
- External resource files (e.g., csv data files, Rmd files, templates)
- Packages required to run the analysis
- Dependencies on previously run analyses
The user also provides a list of "artefacts" (file-based results) that they will produce.
- creates a new empty directory
- copies over only the declared file resources
- loads only the declared packages
- loads the declared R sources
- evaluates any sql queries to create R objects
- then runs the analysis
- verifies that the declared artefacts are produced
It then stores metadata alongside the analysis including md5 hashes of all inputs and outputs, copies of data extracted from the database, a record of all R packages loaded at the end of the session, and (if using git) information about the git state (hash, branch and status).
Then if one of the dependencies of a report changes (the used data, code, etc), we have metadata that can be queried to identify the likely source of the change.
In the MRC Centre for Global Infectious Disease Analysis we use
orderly on two major projects:
- The Vaccine Impact Modelling Consortium
- Our part of the response to the 2018-2019 Ebola outbreak in the Democratic Republic of Congo
The workflows we have developed here are oriented towards collaborative groups of researchers - other workflows are possible (indeed
orderly is also designed to support a decentralised workflow, though this has not been used in practice yet).
In these projects we have a group of researchers who develop and test analyses locally. These are developed on a branch in git and then run on a centralised staging environment (a duplicate of our production environment). The code and outputs are reviewed with the help of GitHub's "Pull requests" and then the reports are run on our production environment.
Interaction with the remote environments is achieved using an HTTP API which
orderly itself transparently uses, so that reports can be run remotely, directly from R. The remote systems also include an interactive web interface that can be used to explore and download versions of analyses, as well as run new ones.
Internal database schema
orderly has a database, which should be the preferred way of querying the report archive from other programs. The schema is programmatically described at
inst/database/schema.yml and automatically generated database documentation is available here.
There is a set of regression tests that require the reference data. Enable these by running the script
./scripts/copy_reference which creates data in
orderly from CRAN with
To install our internally released version (which might be ahead of CRAN) via drat, use
# install.packages("drat") drat:::add("vimc") install.packages("orderly")
MIT © Imperial College of Science, Technology and Medicine
Functions in orderly
|orderly_rebuild||Rebuild the report database|
|orderly_migrate||Migrate an orderly archive|
|orderly_log_on||Orderly logging and diagnostic messages|
|orderly_new||Create new report|
|orderly_pull_dependencies||Download dependent reports|
|orderly_run_info||Information on current orderly run|
|orderly_run||Run a report|
|orderly_commit||Commit a generated report|
|orderly_run_remote||Run a report on a remote server|
|orderly_remote_path||Orderly remote at a different path|
|orderly_test_start||Prepare a directory for orderly to use|
|orderly_db||Connect to orderly databases|
|orderly_deduplicate||Deduplicate an orderly archive|
|orderly_default_remote_set||Set default remote location|
|orderly_example||Set up an orderly example|
|orderly_list_drafts||List draft and archived reports|
|orderly_list||List orderly reports|
|orderly_latest||Find most recent report|
|orderly_init||Initialise an orderly store|
Vignettes of orderly
Last month downloads
|License||MIT + file LICENSE|
|Packaged||2020-01-12 13:07:22 UTC; rich|
|Date/Publication||2020-01-12 14:40:02 UTC|
|imports||DBI , digest , docopt , fs (>= 1.2.7) , ids , R6 , RSQLite , withr , yaml , zip (>= 2.0.0)|
|suggests||httr , jsonlite , knitr , mockery , processx , rmarkdown , testthat , vaultr (>= 1.0.0)|
|Contributors||Robert Ashton, Wes Hinsley, Imperial College of Science, Technology and Medicine , Alex Hill, Martin Eden, Emma Russell, James Thompson|
Include our badge in your README