The drake R package
Data analysis can be slow. A round of scientific computation can take several minutes, hours, or even days to complete. After it finishes, if you update your code or data, your hard-earned results may no longer be valid. How much of that valuable output can you keep, and how much do you need to update? How much runtime must you endure all over again?
For projects in R, the drake
package can help. It analyzes your
workflow, skips
steps with up-to-date results, and orchestrates the rest with optional
distributed
computing. At the
end, drake
provides evidence that your results match the underlying
code and data, which increases your ability to trust your research.
6-minute video
Visit the first page of the manual to watch a short introduction.
What gets done stays done.
Too many data science projects follow a Sisyphean loop:
- Launch the code.
- Wait while it runs.
- Discover an issue.
- Rerun from scratch.
Ordinarily, it is hard to avoid rerunning the code from scratch.
But with drake
, you can automatically
- Launch the parts that changed since last time.
- Skip the rest.
How it works
To set up a project, load your packages,
library(drake)
library(dplyr)
library(ggplot2)
#> Registered S3 methods overwritten by 'ggplot2':
#> method from
#> [.quosures rlang
#> c.quosures rlang
#> print.quosures rlang
load your custom functions,
create_plot <- function(data) {
ggplot(data, aes(x = Petal.Width, fill = Species)) +
geom_histogram()
}
check any supporting files (optional),
# Get the files with drake_example("main").
file.exists("raw_data.xlsx")
#> [1] TRUE
file.exists("report.Rmd")
#> [1] TRUE
and plan what you are going to do.
plan <- drake_plan(
raw_data = readxl::read_excel(file_in("raw_data.xlsx")),
data = raw_data %>%
mutate(Species = forcats::fct_inorder(Species)),
hist = create_plot(data),
fit = lm(Sepal.Width ~ Petal.Width + Species, data),
report = rmarkdown::render(
knitr_in("report.Rmd"),
output_file = file_out("report.html"),
quiet = TRUE
)
)
plan
#> # A tibble: 5 x 2
#> target command
#> <chr> <expr>
#> 1 raw_data readxl::read_excel(file_in("raw_data.xlsx")) …
#> 2 data raw_data %>% mutate(Species = forcats::fct_inorder(Species)) …
#> 3 hist create_plot(data) …
#> 4 fit lm(Sepal.Width ~ Petal.Width + Species, data) …
#> 5 report rmarkdown::render(knitr_in("report.Rmd"), output_file = file_ou…
So far, we have just been setting the stage. Use make()
to do the real
work. Targets are built in the correct order regardless of the row order
of plan
.
make(plan)
#> target raw_data
#> target data
#> target fit
#> target hist
#> target report
Except for files like report.html
, your output is stored in a hidden
.drake/
folder. Reading it back is easy.
readd(data) # See also loadd().
#> # A tibble: 150 x 5
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> <dbl> <dbl> <dbl> <dbl> <fct>
#> 1 5.1 3.5 1.4 0.2 setosa
#> 2 4.9 3 1.4 0.2 setosa
#> 3 4.7 3.2 1.3 0.2 setosa
#> 4 4.6 3.1 1.5 0.2 setosa
#> 5 5 3.6 1.4 0.2 setosa
#> # … with 145 more rows
You may look back on your work and see room for improvement, but it’s
all good! The whole point of drake
is to help you go back and change
things quickly and painlessly. For example, we forgot to give our
histogram a bin width.
readd(hist)
#> `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
So let’s fix the plotting function.
create_plot <- function(data) {
ggplot(data, aes(x = Petal.Width, fill = Species)) +
geom_histogram(binwidth = 0.25) +
theme_gray(20)
}
drake
knows which results are affected.
config <- drake_config(plan)
vis_drake_graph(config) # Interactive graph: zoom, drag, etc.
The next make()
just builds hist
and report.html
. No point in
wasting time on the data or model.
make(plan)
#> target hist
#> target report
loadd(hist)
hist
Reproducibility with confidence
The R community emphasizes reproducibility. Traditional themes include
scientific
replicability,
literate programming with knitr, and
version control with
git.
But internal consistency is important too. Reproducibility carries the
promise that your output matches the code and data you say you used.
With the exception of non-default
triggers and
hasty
mode,
drake
strives to keep this promise.
Evidence
Suppose you are reviewing someone else’s data analysis project for
reproducibility. You scrutinize it carefully, checking that the datasets
are available and the documentation is thorough. But could you re-create
the results without the help of the original author? With drake
, it is
quick and easy to find out.
make(plan)
#> All targets are already up to date.
config <- drake_config(plan)
outdated(config)
#> character(0)
With everything already up to date, you have tangible evidence of reproducibility. Even though you did not re-create the results, you know the results are re-creatable. They faithfully show what the code is producing. Given the right package environment and system configuration, you have everything you need to reproduce all the output by yourself.
Ease
When it comes time to actually rerun the entire project, you have much more confidence. Starting over from scratch is trivially easy.
clean() # Remove the original author's results.
make(plan) # Independently re-create the results from the code and input data.
#> target raw_data
#> target data
#> target fit
#> target hist
#> target report
Independent replication
With even more evidence and confidence, you can invest the time to
independently replicate the original code base if necessary. Up until
this point, you relied on basic drake
functions such as make()
, so
you may not have needed to peek at any substantive author-defined code
in advance. In that case, you can stay usefully ignorant as you
reimplement the original author’s methodology. In other words, drake
could potentially improve the integrity of independent replication.
Readability and transparency
Ideally, independent observers should be able to read your code and
understand it. drake
helps in several ways.
- The workflow plan data
frame
explicitly outlines the steps of the analysis, and
vis_drake_graph()
visualizes how those steps depend on each other. drake
takes care of the parallel scheduling and high-performance computing (HPC) for you. That means the HPC code is no longer tangled up with the code that actually expresses your ideas.- You can generate large collections of targets without necessarily changing your code base of imported functions, another nice separation between the concepts and the execution of your workflow
Aggressively scale up.
Not every project can complete in a single R session on your laptop.
Some projects need more speed or computing power. Some require a few
local processor cores, and some need large high-performance computing
systems. But parallel computing is hard. Your tables and figures depend
on your analysis results, and your analyses depend on your datasets, so
some tasks must finish before others even begin. drake
knows what to
do. Parallelism is implicit and automatic. See the high-performance
computing guide
for all the details.
# Use the spare cores on your local machine.
make(plan, jobs = 4)
# Or scale up to a supercomputer.
drake_hpc_template_file("slurm_clustermq.tmpl") # https://slurm.schedmd.com/
options(
clustermq.scheduler = "clustermq",
clustermq.template = "slurm_clustermq.tmpl"
)
make(plan, parallelism = "clustermq", jobs = 4)
Installation
You can choose among different versions of drake
. The CRAN release
often lags behind the online
manual but may have fewer
bugs.
# Install the latest stable release from CRAN.
install.packages("drake")
# Alternatively, install the development version from GitHub.
install.packages("devtools")
library(devtools)
install_github("ropensci/drake")
Function reference
The reference section lists all the available functions. Here are the most important ones.
drake_plan()
: create a workflow data frame (likemy_plan
).make()
: build your project.r_make()
: launch a freshcallr::r()
process to build your project. Called from an interactive R session,r_make()
is more reproducible thanmake()
.loadd()
: load one or more built targets into your R session.readd()
: read and return a built target.drake_config()
: create a master configuration list for other user-side functions.vis_drake_graph()
: show an interactive visual network representation of your workflow.outdated()
: see which targets will be built in the nextmake()
.deps()
: check the dependencies of a command or function.failed()
: list the targets that failed to build in the lastmake()
.diagnose()
: return the full context of a build, including errors, warnings, and messages.
Documentation
- The user manual
- The reference website.
- The official repository of example
code. Download an
example workflow from here with
drake_example()
. drakeplanner
, an R/Shiny app to help learndrake
and create new projects. Run locally withdrakeplanner::drakeplanner()
or access it at https://wlandau.shinyapps.io/drakeplanner.- Presentations and workshops by Will Landau, Kirill Müller, Amanda Dobbyn, Karthik Ram, Sina Rüeger, Christine Stawitz, and others. See specific links at https://ropenscilabs.github.io/drake-manual/index.html#presentations
- The FAQ page, which links to appropriately-labeled issues on GitHub.
Use cases
The official rOpenSci use cases and
associated discussion threads
describe applications of drake
in action. Here are some more
applications of drake
in real-world
projects.
- efcaguab/demografia-del-voto
- efcaguab/great-white-shark-nsw
- IndianaCHE/Detailed-SSP-Reports
- pat-s/pathogen-modeling
- sol-eng/tensorflow-w-r
- tiernanmartin/home-and-hope
Help and troubleshooting
The following resources document many known issues and challenges.
- Frequently-asked questions.
- Cautionary notes and edge cases
- Debugging and testing drake projects
- Other known issues (please search both open and closed ones).
If you are still having trouble, please submit a new issue with a bug report or feature request, along with a minimal reproducible example where appropriate.
The GitHub issue tracker is mainly intended for bug reports and feature
requests. While questions about usage etc. are also highly encouraged,
you may alternatively wish to post to Stack
Overflow and use the drake-r-package
tag.
Contributing
Development is a community effort, and we encourage participation. Please read CONTRIBUTING.md for details.
Similar work
GNU Make
The original idea of a time-saving reproducible build system extends back at least as far as GNU Make, which still aids the work of data scientists as well as the original user base of complied language programmers. In fact, the name “drake” stands for “Data Frames in R for Make”. Make is used widely in reproducible research. Below are some examples from Karl Broman’s website.
- Bostock, Mike (2013). “A map of flowlines from NHDPlus.” https://github.com/mbostock/us-rivers. Powered by the Makefile at https://github.com/mbostock/us-rivers/blob/master/Makefile.
- Broman, Karl W (2012). “Halotype Probabilities in Advanced
Intercross Populations.” G3 2(2), 199-202.Powered by the
Makefile
at https://github.com/kbroman/ailProbPaper/blob/master/Makefile. - Broman, Karl W (2012). “Genotype Probabilities at Intermediate Generations in the Construction of Recombinant Inbred Lines.” *Genetics 190(2), 403-412. Powered by the Makefile at https://github.com/kbroman/preCCProbPaper/blob/master/Makefile.
- Broman, Karl W and Kim, Sungjin and Sen, Saunak and Ane, Cecile and
Payseur, Bret A (2012). “Mapping Quantitative Trait Loci onto a
Phylogenetic Tree.” Genetics 192(2), 267-279. Powered by the
Makefile
at https://github.com/kbroman/phyloQTLpaper/blob/master/Makefile.
There are several reasons for R users to prefer drake
instead.
drake
already has a Make-powered parallel backend. Just runmake(..., parallelism = "Makefile", jobs = 2)
to enjoy most of the original benefits of Make itself.- Improved scalability. With
Make, you must write a
potentially large and cumbersome
Makefile
by hand. But with
drake
, you can use wildcard templating to automatically generate massive collections of targets with minimal code. - Lower overhead for light-weight tasks. For each
Make target that uses R, a
brand new R session must spawn. For projects with thousands of small
targets, that means more time may be spent loading R sessions than
doing the actual work. With
make(..., parallelism = "mclapply, jobs = 4")
,drake
launches 4 persistent workers up front and efficiently processes the targets in R. - Convenient organization of output. With
Make, the user must save each
target as a file.
drake
saves all the results for you automatically in a storr cache so you do not have to micromanage the results.
Remake
drake overlaps with its direct predecessor, remake. In fact, drake owes its core ideas to remake and Rich FitzJohn. Remake’s development repository lists several real-world applications. drake surpasses remake in several important ways, including but not limited to the following.
- High-performance computing.
Remake
has no native parallel computing support.drake
, on the other hand, has a thorough selection of parallel computing technologies and scheduling algorithms. Thanks tofuture
,future.batchtools
, andbatchtools
, it is straightforward to configure adrake
project for most popular job schedulers, such as SLURM, TORQUE, and the Grid Engine, as well as systems contained in Docker images. - A friendly interface. In
remake
, the user must manually write a YAML configuration file to arrange the steps of a workflow, which leads to some of the same scalability problems as Make.drake
’s domain-specific language easily generates workflows at scale. - Thorough documentation.
drake
contains thorough user manual, a reference website, a comprehensive README, examples in the help files of user-side functions, and accessible example code that users can write withdrake::example_drake()
. - Active maintenance.
drake
is actively developed and maintained, and issues are usually addressed promptly. - Presence on CRAN. At the time of writing,
drake
is available on CRAN, butremake
is not.
Memoise
Memoization is the strategic caching of the return values of functions. Every time a memoized function is called with a new set of arguments, the return value is saved for future use. Later, whenever the same function is called with the same arguments, the previous return value is salvaged, and the function call is skipped to save time. The memoise package is an excellent implementation of memoization in R.
However, memoization does not go far enough. In reality, the return
value of a function depends not only on the function body and the
arguments, but also on any nested functions and global variables, the
dependencies of those dependencies, and so on upstream. drake
surpasses memoise because it uses
the entire dependency network graph of a project to decide which
pieces need to be rebuilt and which ones can be skipped.
Knitr
Much of the R community uses knitr for
reproducible research. The idea is to intersperse code chunks in an R
Markdown or *.Rnw
file and then
generate a dynamic report that weaves together code, output, and prose.
Knitr is not designed to be a serious
pipeline toolkit, and
it should not be the primary computational engine for medium to large
data analysis projects.
- Knitr scales far worse than Make or remake. The whole point is to consolidate output and prose, so it deliberately lacks the essential modularity.
- There is no obvious high-performance computing support.
- While there is a way to skip chunks that are already up to date
(with code chunk options
cache
andautodep
), this functionality is not the focus of knitr. It is deactivated by default, and remake anddrake
are more dependable ways to skip work that is already up to date.
drake
was designed to manage the entire workflow with
knitr reports as targets. The strategy is
analogous for knitr reports within
remake projects.
Factual’s Drake
Factual’s Drake is similar in concept, but the development effort is completely unrelated to the drake R package.
Other pipeline toolkits
There are countless other successful pipeline
toolkits. The drake
package distinguishes itself with its R-focused approach,
Tidyverse-friendly interface, and a thorough selection of parallel
computing technologies and scheduling
algorithms.
Acknowledgements
Special thanks to Jarad Niemi, my advisor from
graduate school, for first introducing me
to the idea of Makefiles for
research. He originally set me down the path that led to drake
.
Many thanks to Julia Lowndes, Ben Marwick, and Peter Slaughter for reviewing drake for rOpenSci, and to Maëlle Salmon for such active involvement as the editor. Thanks also to the following people for contributing early in development.
- Alex Axthelm
- Chan-Yub Park
- Daniel Falster
- Eric Nantz
- Henrik Bengtsson
- Ian Watson
- Jasper Clarkberg
- Kendon Bell
- Kirill Müller
- Michael Schubert
Credit for images is attributed here.