dplyr v0.4.2
Monthly downloads
A Grammar of Data Manipulation
A fast, consistent tool for working with data frame like objects,
both in memory and out of memory.
Readme
dplyr
dplyr is the next iteration of plyr, focussed on tools for working with data frames (hence the d
in the name). It has three main goals:
Identify the most important data manipulation tools needed for data analysis and make them easy to use from R.
Provide blazing fast performance for in-memory data by writing key pieces in C++.
Use the same interface to work with data no matter where it's stored, whether in a data frame, a data table or database.
You can install:
the latest released version from CRAN with
install.packages("dplyr")
the latest development version from github with
if (packageVersion("devtools") < 1.6) { install.packages("devtools") } devtools::install_github("hadley/lazyeval") devtools::install_github("hadley/dplyr")
You'll probably also want to install the data packages used in most examples: install.packages(c("nycflights13", "Lahman"))
.
If you encounter a clear bug, please file a minimal reproducible example on github. For questions and other discussion, please use the manipulatr mailing list.
Learning dplyr
To get started, read the notes below, then read the intro vignette: vignette("introduction", package = "dplyr")
. To make the most of dplyr, I also recommend that you familiarise yourself with the principles of tidy data: this will help you get your data into a form that works well with dplyr, ggplot2 and R's many modelling functions.
If you need more, help I recommend the following (paid) resources:
dplyr on datacamp, by Garrett Grolemund. Learn the basics of dplyr at your own pace in this interactive online course.
Introduction to Data Science with R: How to Manipulate, Visualize, and Model Data with the R Language, by Garrett Grolemund. This O'Reilly video series will teach you the basics needed to be an effective analyst in R.
Key data structures
The key object in dplyr is a tbl, a representation of a tabular data structure. Currently dplyr
supports:
- data frames
- data tables
- SQLite
- PostgreSQL/Redshift
- MySQL/MariaDB
- Bigquery
- MonetDB
- data cubes with arrays (partial implementation)
You can create them as follows:
library(dplyr) # for functions
library(nycflights13) # for data
flights
#> Source: local data frame [336,776 x 16]
#>
#> year month day dep_time dep_delay arr_time arr_delay carrier tailnum
#> 1 2013 1 1 517 2 830 11 UA N14228
#> 2 2013 1 1 533 4 850 20 UA N24211
#> 3 2013 1 1 542 2 923 33 AA N619AA
#> 4 2013 1 1 544 -1 1004 -18 B6 N804JB
#> 5 2013 1 1 554 -6 812 -25 DL N668DN
#> 6 2013 1 1 554 -4 740 12 UA N39463
#> 7 2013 1 1 555 -5 913 19 B6 N516JB
#> 8 2013 1 1 557 -3 709 -14 EV N829AS
#> 9 2013 1 1 557 -3 838 -8 B6 N593JB
#> 10 2013 1 1 558 -2 753 8 AA N3ALAA
#> .. ... ... ... ... ... ... ... ... ...
#> Variables not shown: flight (int), origin (chr), dest (chr), air_time
#> (dbl), distance (dbl), hour (dbl), minute (dbl)
# Caches data in local SQLite db
flights_db1 <- tbl(nycflights13_sqlite(), "flights")
# Caches data in local postgres db
flights_db2 <- tbl(nycflights13_postgres(), "flights")
Each tbl also comes in a grouped variant which allows you to easily perform operations "by group":
carriers_df <- flights %>% group_by(carrier)
carriers_db1 <- flights_db1 %>% group_by(carrier)
carriers_db2 <- flights_db2 %>% group_by(carrier)
Single table verbs
dplyr
implements the following verbs useful for data manipulation:
select()
: focus on a subset of variablesfilter()
: focus on a subset of rowsmutate()
: add new columnssummarise()
: reduce each group to a smaller number of summary statisticsarrange()
: re-order the rows
They all work as similarly as possible across the range of data sources. The main difference is performance:
system.time(carriers_df %>% summarise(delay = mean(arr_delay)))
#> user system elapsed
#> 0.036 0.001 0.037
system.time(carriers_db1 %>% summarise(delay = mean(arr_delay)) %>% collect())
#> user system elapsed
#> 0.263 0.130 0.392
system.time(carriers_db2 %>% summarise(delay = mean(arr_delay)) %>% collect())
#> user system elapsed
#> 0.016 0.001 0.151
Data frame methods are much much faster than the plyr equivalent. The database methods are slower, but can work with data that don't fit in memory.
system.time(plyr::ddply(flights, "carrier", plyr::summarise,
delay = mean(arr_delay, na.rm = TRUE)))
#> user system elapsed
#> 0.100 0.032 0.133
do()
As well as the specialised operations described above, dplyr
also provides the generic do()
function which applies any R function to each group of the data.
Let's take the batting database from the built-in Lahman database. We'll group it by year, and then fit a model to explore the relationship between their number of at bats and runs:
by_year <- lahman_df() %>%
tbl("Batting") %>%
group_by(yearID)
by_year %>%
do(mod = lm(R ~ AB, data = .))
#> Source: local data frame [143 x 2]
#> Groups: <by row>
#>
#> yearID mod
#> 1 1871 <S3:lm>
#> 2 1872 <S3:lm>
#> 3 1873 <S3:lm>
#> 4 1874 <S3:lm>
#> 5 1875 <S3:lm>
#> 6 1876 <S3:lm>
#> 7 1877 <S3:lm>
#> 8 1878 <S3:lm>
#> 9 1879 <S3:lm>
#> 10 1880 <S3:lm>
#> .. ... ...
Note that if you are fitting lots of linear models, it's a good idea to use biglm
because it creates model objects that are considerably smaller:
by_year %>%
do(mod = lm(R ~ AB, data = .)) %>%
object.size() %>%
print(unit = "MB")
#> 22.2 Mb
by_year %>%
do(mod = biglm::biglm(R ~ AB, data = .)) %>%
object.size() %>%
print(unit = "MB")
#> 0.8 Mb
Multiple table verbs
As well as verbs that work on a single tbl, there are also a set of useful verbs that work with two tbls at a time: joins and set operations.
dplyr implements the four most useful joins from SQL:
inner_join(x, y)
: matching x + yleft_join(x, y)
: all x + matching ysemi_join(x, y)
: all x with match in yanti_join(x, y)
: all x without match in y
And provides methods for:
intersect(x, y)
: all rows in both x and yunion(x, y)
: rows in either x or ysetdiff(x, y)
: rows in x, but not y
Plyr compatibility
You'll need to be a little careful if you load both plyr and dplyr at the same time. I'd recommend loading plyr first, then dplyr, so that the faster dplyr functions come first in the search path. By and large, any function provided by both dplyr and plyr works in a similar way, although dplyr functions tend to be faster and more general.
Related approaches
Functions in dplyr
Name | Description | |
arrange | Arrange rows by variables. | |
bench_compare | Evaluate, compare, benchmark operations of a set of srcs. | |
backend_sql | SQL generation. | |
make_tbl | Create a "tbl" object | |
id | Compute a unique numeric id for each unique row in a data frame. | |
between | Do values in a numeric vector fall in specified range? | |
tbl_sql | Create an SQL tbl (abstract) | |
build_sql | Build a SQL string. | |
chain | Chain together multiple operations. | |
cumall | Cumulativate versions of any, all, and mean | |
grouped_df | A grouped data frame. | |
as.tbl_cube | Coerce an existing data structure into a | |
ranking | Windowed rank functions. | |
group_by | Group a tbl by one or more variables. | |
tbl | Create a table from a data source | |
select_vars | Select variables. | |
compute | Compute a lazy tbl. | |
src | Create a "src" object | |
join.tbl_sql | Join sql tbls. | |
group_indices | Group id. | |
n_distinct | Efficiently count the number of unique values in a vector. | |
rowwise | Group input by rows | |
desc | Descending order. | |
partial_eval | Partially evaluate an expression. | |
nth | Extract the first, last or nth value from a vector. | |
dplyr | dplyr: a grammar of data manipulation | |
top_n | Select top n rows (by value). | |
data_frame | Build a data frame. | |
funs | Create a list of functions calls. | |
sample | Sample n rows from a table. | |
base_scalar | Create an sql translator | |
backend_src | Source generics. | |
tally | Counts/tally observations by group. | |
temp_srcs | Connect to temporary data sources. | |
backend_db | Database generics. | |
n | The number of observations in the current group. | |
with_order | Run a function with one order, translating result back to original order | |
tbl_cube | A data cube tbl. | |
type_sum | Provide a succint summary of a type | |
tbl_dt | Create a data table tbl. | |
sql | SQL escaping. | |
query | Create a mutable query object. | |
summarise_each | Summarise and mutate multiple columns. | |
tbl_df | Create a data frame tbl. | |
same_src | Figure out if two sources are the same (or two tbl have the same source) | |
knit_print.trunc_mat | knit_print method for trunc mat | |
src_postgres | Connect to postgresql. | |
src_local | A local source. | |
src_sqlite | Connect to a sqlite database. | |
bind | Efficiently bind multiple data frames by row and column. | |
dplyr-cluster | Cluster management. | |
translate_sql | Translate an expression to sql. | |
copy_to | Copy a local data frame to a remote src. | |
summarise | Summarise multiple values to a single value. | |
grouped_dt | A grouped data table. | |
join | Join two tbls together. | |
location | Print the location in memory of a data frame | |
tbl_vars | List variables provided by a tbl. | |
glimpse | Get a glimpse of your data. | |
nasa | NASA spatio-temporal data | |
order_by | A helper function for ordering window function output. | |
slice | Select rows by position. | |
explain | Explain details of an tbl. | |
copy_to.src_sql | Copy a local data frame to a sqlite src. | |
distinct | Select distinct/unique rows. | |
select | Select/rename variables by name. | |
join.tbl_df | Join data frame tbls. | |
progress_estimated | Progress bar with estimated time. | |
lahman | Cache and retrieve an | |
sql_quote | Helper function for quoting sql elements. | |
add_rownames | Convert row names to an explicit variable. | |
setops | Set operations. | |
filter | Return rows with matching conditions. | |
group_size | Calculate group sizes. | |
nycflights13 | Database versions of the nycflights13 data | |
as_data_frame | Coerce a list to a data frame. | |
mutate | Add new variables. | |
src_sql | Create a "sql src" object | |
all.equal.tbl_df | Provide a useful implementation of all.equal for data.frames. | |
do | Do arbitrary operations on a tbl. | |
lead-lag | Lead and lag. | |
failwith | Fail with specified value. | |
print.tbl_df | Tools for describing matrices | |
groups | Get/set the grouping variables for tbl. | |
src_mysql | Connect to mysql/mariadb. | |
join.tbl_dt | Join data table tbls. | |
src_tbls | List all tbls provided by a source. | |
No Results! |
Last month downloads
Details
Type | Package |
URL | https://github.com/hadley/dplyr |
BugReports | https://github.com/hadley/dplyr/issues |
VignetteBuilder | knitr |
LazyData | yes |
LinkingTo | Rcpp (>= 0.11.6), BH (>= 1.58.0-1) |
License | MIT + file LICENSE |
Collate | 'RcppExports.R' 'all-equal.r' 'bench-compare.r' 'chain.r' 'cluster.R' 'colwise.R' 'compute-collect.r' 'copy-to.r' 'data-lahman.r' 'data-nasa.r' 'data-nycflights13.r' 'data-temp.r' 'data.r' 'dataframe.R' 'dbi-s3.r' 'desc.r' 'distinct.R' 'do.r' 'dplyr.r' 'explain.r' 'failwith.r' 'funs.R' 'glimpse.R' 'group-by.r' 'group-indices.R' 'group-size.r' 'grouped-df.r' 'grouped-dt.r' 'id.r' 'inline.r' 'join.r' 'lead-lag.R' 'location.R' 'manip.r' 'nth-value.R' 'order-by.R' 'over.R' 'partial-eval.r' 'progress.R' 'query.r' 'rank.R' 'rbind.r' 'rowwise.r' 'sample.R' 'select-utils.R' 'select-vars.R' 'sets.r' 'sql-escape.r' 'sql-star.r' 'src-local.r' 'src-mysql.r' 'src-postgres.r' 'src-sql.r' 'src-sqlite.r' 'src.r' 'tally.R' 'tbl-cube.r' 'tbl-df.r' 'tbl-dt.r' 'tbl-sql.r' 'tbl.r' 'top-n.R' 'translate-sql-helpers.r' 'translate-sql-base.r' 'translate-sql-window.r' 'translate-sql.r' 'type-sum.r' 'utils-dt.R' 'utils-format.r' 'utils.r' 'view.r' 'zzz.r' |
imports | assertthat , DBI (>= 0.3) , lazyeval (>= 0.1.10) , magrittr , R6 , Rcpp , utils |
depends | base (>= 3.1.2) , R (>= 3.1.2) |
linkingto | BH (>= 1.58.0-1) |
suggests | data , data.table , ggplot2 , knitr , Lahman (>= 3.0-1) , methods , mgcv , microbenchmark , nycflights13 , RMySQL , RPostgreSQL , RSQLite (>= 1.0.0) , testthat |
Contributors | Romain Francois, RStudio |
Include our badge in your README
[](http://www.rdocumentation.org/packages/dplyr)