dplyr v0.2


Monthly downloads



by Hadley Wickham

dplyr: a grammar of data manipulation

A fast, consistent tool for working with data frame like objects, both in memory and out of memory.



Build Status

dplyr is the next iteration of plyr, focussed on tools for working with data frames (hence the d in the name). It has three main goals:

  • Identify the most important data manipulation tools needed for data analysis and make them easy to use from R.

  • Provide blazing fast performance for in-memory data by writing key pieces in C++.

  • Use the same interface to work with data no matter where it's stored, whether in a data frame, a data table or database.

You can install:

  • the latest released version from CRAN with

  • the latest development version from github with


To get started, read the notes below, then read the intro vignette: vignette("introduction", package = "dplyr"). To make the most of dplyr, I also recommend that you familiarise yourself with the principles of tidy data: this will help you get your data into a form that works well with dplyr, ggplot2 and R's many modelling functions.

If you encounter a clear bug, please file a minimal reproducible example on github. For questions and other discussion, please use the manipulatr mailing list.


The key object in dplyr is a tbl, a representation of a tabular data structure. Currently dplyr supports:

You can create them as follows:

library(dplyr) # for functions
library(hflights) # for data

# Coerce to data table
hflights_dt <- tbl_dt(hflights)

# Caches data in local SQLite db
hflights_db1 <- tbl(hflights_sqlite(), "hflights")

# Caches data in local postgres db
hflights_db2 <- tbl(hflights_postgres(), "hflights")

Each tbl also comes in a grouped variant which allows you to easily perform operations "by group":

carriers_df  <- group_by(hflights, UniqueCarrier)
carriers_dt  <- group_by(hflights_dt, UniqueCarrier)
carriers_db1 <- group_by(hflights_db1, UniqueCarrier)
carriers_db2 <- group_by(hflights_db2, UniqueCarrier)

Single table verbs

dplyr implements the following verbs useful for data manipulation:

  • select(): focus on a subset of variables
  • filter(): focus on a subset of rows
  • mutate(): add new columns
  • summarise(): reduce each group to a smaller number of summary statistics
  • arrange(): re-order the rows

See ?manip for more details.

They all work as similarly as possible across the range of data sources. The main difference is performance:

system.time(summarise(carriers_df, delay = mean(ArrDelay, na.rm = TRUE)))
#   user  system elapsed
#  0.010   0.002   0.012
system.time(summarise(carriers_dt, delay = mean(ArrDelay, na.rm = TRUE)))
#   user  system elapsed
#  0.007   0.000   0.008
system.time(summarise(collect(carriers_db1, delay = mean(ArrDelay))))
#   user  system elapsed
#  0.402   0.058   0.465
system.time(summarise(collect(carriers_db2, delay = mean(ArrDelay))))
#   user  system elapsed
#  0.386   0.097   0.718

The data frame and data table methods are order of magnitude faster than plyr. The database methods are slower, but can work with data that don't fit in memory.

system.time(ddply(hflights, "UniqueCarrier", summarise,
  delay = mean(ArrDelay, na.rm = TRUE)))
#   user  system elapsed
#  0.527   0.078   0.604


As well as the specialised operations described above, dplyr also provides the generic do() function which applies any R function to each group of the data.

Let's take the batting database from the built-in Lahman database. We'll group it by year, and then fit a model to explore the relationship between their number of at bats and runs:

batting_db <- tbl(lahman_sqlite(), "Batting")
batting_df <- collect(batting_db)
batting_dt <- tbl_dt(batting_df)

years_db <- group_by(batting_db, yearID)
years_df <- group_by(batting_df, yearID)
years_dt <- group_by(batting_dt, yearID)

system.time(do(years_db, failwith(NULL, lm), formula = R ~ AB))
system.time(do(years_df, failwith(NULL, lm), formula = R ~ AB))
system.time(do(years_dt, failwith(NULL, lm), formula = R ~ AB))

Note that if you are fitting lots of linear models, it's a good idea to use biglm because it creates model objects that are considerably smaller:

mod1 <- do(years_df, lm, formula = R ~ AB)
mod2 <- do(years_df, biglm, formula = R ~ AB)
print(object.size(mod1), unit = "MB")
print(object.size(mod2), unit = "MB")

Binary verbs

As well as verbs that work on a single tbl, there are also a set of useful verbs that work with two tbls are a time: joins. dplyr implements the four most useful joins from SQL:

  • inner_join(x, y): matching x + y
  • left_join(x, y): all x + matching y
  • semi_join(x, y): all x with match in y
  • anti_join(x, y): all x without match in y

Currently join variables must be the same in both the left-hand and right-hand sides.

Other operations

All tbls also provide head() and print() methods. The default print method gives information about the data source and shows the first 10 rows and all the columns that will fit on one screen.

Plyr compatibility

You'll need to be a little careful if you load both plyr and dplyr at the same time. I'd recommend loading plyr first, then dplyr, so that the faster dplyr functions come first in the search path. By and large, any function provided by both dplyr and plyr works in a similar way, although dplyr functions tend to be faster and more general.

Functions in dplyr

Name Description
join.tbl_df Join data table tbls.
rowwise Group input by rows
src_bigquery A bigquery data source.
tbl Create a table from a data source
src_sql Create a "sql src" object
copy_to.src_sql Copy a local data fram to a sqlite src.
hflights_df Database versions of the hflights data
id Compute a unique numeric id for each unique row in a data frame.
sample Sample n rows from a table.
join.tbl_sql Join sql tbls.
top_n Select top n rows (by value).
location Print the location in memory of a data frame
with_order Run a function with one order, translating result back to original order
bench_compare Evaluate, compare, benchmark operations of a set of srcs.
dplyr The dplyr package.
desc Descending order.
select_vars Select variables.
explain_sql Show sql and query plans.
group_by Group a tbl by one or more variables.
cumall Cumulativate versions of any, all, and mean
make_tbl Create a "tbl" object
src_postgres Connect to postgresql.
join Join two tbls together.
n_distinct Efficiently count the number of unique values in a vector.
tbl_cube A data cube tbl.
temp_srcs Connect to temporary data sources.
manip_grouped_dt Data manipulation for grouped data tables.
manip_df Data manipulation for data frames.
tally Tally observations by group.
n The number of observations in the current group.
rbind_all Efficiently rbind multiple data frames.
type_sum Provide a succint summary of a type
src_tbls List all tbls provided by a source.
failwith Fail with specified value.
grouped_dt A grouped data table.
print.tbl_df Tools for describing matrices
src Create a "src" object
partial_eval Partially evaluate an expression.
translate_sql Translate an expression to sql.
sql SQL escaping.
groups Get/set the grouping variables for tbl.
summarise_each Summarise and mutate multiple columns.
ranking Windowed rank functions.
nth Extract the first, last or nth value from a vector.
same_src Figure out if two sources are the same (or two tbl have the same source)
setops Set operations.
Progress-class Progress bar with estimated time.
as.tbl_cube Coerce an existing data structure into a
dplyr-cluster Cluster management.
grouped_df A grouped data frame.
tbl_vars List variables provided by a tbl.
group_size Calculate the size of each group
join.tbl_dt Join data table tbls.
funs Create a list of functions calls.
manip_dt Data manipulation for data tables.
src_local A local source.
setops-data.frame Set operations for data frames.
lead-lag Lead and lag.
manip Data manipulation functions.
src_sqlite Connect to a sqlite database.
order_by A helper function for ordering window function output.
compute Compute a lazy tbl.
tbl_df Create a data frame tble.
tbl_sql Create an SQL tbl (abstract)
src_monetdb Connect to MonetDB (http://www.monetdb.org), an Open Source analytics-focused database
build_sql Build a SQL string.
base_scalar Create an sql translator
copy_to Copy a local data frame to a remote src.
glimpse Get a glimpse of your data.
explain Explain details of an object
lahman Cache and retrieve an
tbl_dt Create a data table tbl.
chain Chain together multiple operations.
all.equal.tbl_df Provide a useful implementation of all.equal for data.frames.
do Do arbitrary operations on a tbl.
nasa NASA spatio-temporal data
src_mysql Connect to mysql/mariadb.
query Create a mutable query object.
No Results!

Last month downloads


Type Package
URL https://github.com/hadley/dplyr
Roxygen list(wrap = FALSE)
VignetteBuilder knitr
LazyData yes
LinkingTo Rcpp (>= 0.11.1), BH (>= 1.51.0-2)
License MIT + file LICENSE
Collate 'RcppExports.R' 'all-equal.r' 'base.R' 'bench-compare.r' 'cbind.r' 'chain.r' 'cluster.R' 'colwise.R' 'compute-collect.r' 'copy-to.r' 'data-hflights.r' 'data-lahman.r' 'data-nasa.r' 'data-temp.r' 'data.r' 'dbi-s3.r' 'desc.r' 'do.r' 'dplyr.r' 'explain.r' 'failwith.r' 'funs.R' 'glimpse.R' 'group-by.r' 'group-size.r' 'grouped-df.r' 'grouped-dt.r' 'id.r' 'inline.r' 'join-df.r' 'join-dt.r' 'join-sql.r' 'join.r' 'lead-lag.R' 'location.R' 'manip-cube.r' 'manip-df.r' 'manip-dt.r' 'manip-grouped-dt.r' 'manip-sql.r' 'manip.r' 'nth-value.R' 'order-by.R' 'over.R' 'partial-eval.r' 'progress.R' 'query.r' 'query-bq.r' 'rank.R' 'rbind.r' 'rowwise.r' 'sample.R' 'select-vars.R' 'sets.r' 'sql-escape.r' 'sql-star.r' 'src-bigquery.r' 'src-local.r' 'src-monetdb.r' 'src-mysql.r' 'src-postgres.r' 'src-sql.r' 'src-sqlite.r' 'src.r' 'tally.R' 'tbl-cube.r' 'tbl-data-frame.R' 'tbl-df.r' 'tbl-dt.r' 'tbl-sql.r' 'tbl.r' 'top-n.R' 'translate-sql-helpers.r' 'translate-sql-base.r' 'translate-sql-window.r' 'translate-sql.r' 'type-sum.r' 'utils-dt.R' 'utils-format.r' 'utils.r' 'view.r' 'zzz.r'

Include our badge in your README