dplyr v0.4.3

0

Monthly downloads

0th

Percentile

by Hadley Wickham

A Grammar of Data Manipulation

A fast, consistent tool for working with data frame like objects, both in memory and out of memory.

Readme

dplyr

Build Status

dplyr is the next iteration of plyr. It is focussed on tools for working with data frames (hence the d in its name). It has three main goals:

  • Identify the most important data manipulation tools needed for data analysis and make them easy to use in R.

  • Provide blazing fast performance for in-memory data by writing key pieces of code in C++.

  • Use the same code interface to work with data no matter where it's stored, whether in a data frame, a data table or database.

You can install:

  • the latest released version from CRAN with

    install.packages("dplyr")
    
  • the latest development version from github with

    if (packageVersion("devtools") < 1.6) {
      install.packages("devtools")
    }
    devtools::install_github("hadley/lazyeval")
    devtools::install_github("hadley/dplyr")
    

You'll probably also want to install the data packages used in most examples: install.packages(c("nycflights13", "Lahman")).

If you encounter a clear bug, please file a minimal reproducible example on github. For questions and other discussion, please use the manipulatr mailing list.

Learning dplyr

To get started, read the notes below, and then read the intro vignette: vignette("introduction", package = "dplyr"). To make the most of dplyr, I also recommend that you familiarise yourself with the principles of tidy data: this will help you get your data into a form that works well with dplyr, ggplot2 and R's many modelling functions.

If you need more, help I recommend the following (paid) resources:

  • dplyr on datacamp, by Garrett Grolemund. Learn the basics of dplyr at your own pace in this interactive online course.

  • Introduction to Data Science with R: How to Manipulate, Visualize, and Model Data with the R Language, by Garrett Grolemund. This O'Reilly video series will teach you the basics needed to be an effective analyst in R.

Key data structures

The key object in dplyr is a tbl, a representation of a tabular data structure. Currently dplyr supports:

You can create them as follows:

library(dplyr) # for functions
library(nycflights13) # for data
flights
#> Source: local data frame [336,776 x 16]
#> 
#>     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum
#>    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)
#> 1   2013     1     1      517         2      830        11      UA  N14228
#> 2   2013     1     1      533         4      850        20      UA  N24211
#> 3   2013     1     1      542         2      923        33      AA  N619AA
#> 4   2013     1     1      544        -1     1004       -18      B6  N804JB
#> 5   2013     1     1      554        -6      812       -25      DL  N668DN
#> 6   2013     1     1      554        -4      740        12      UA  N39463
#> 7   2013     1     1      555        -5      913        19      B6  N516JB
#> 8   2013     1     1      557        -3      709       -14      EV  N829AS
#> 9   2013     1     1      557        -3      838        -8      B6  N593JB
#> 10  2013     1     1      558        -2      753         8      AA  N3ALAA
#> ..   ...   ...   ...      ...       ...      ...       ...     ...     ...
#> Variables not shown: flight (int), origin (chr), dest (chr), air_time
#>   (dbl), distance (dbl), hour (dbl), minute (dbl)

# Caches data in local SQLite db
flights_db1 <- tbl(nycflights13_sqlite(), "flights")

# Caches data in local postgres db
flights_db2 <- tbl(nycflights13_postgres(), "flights")

A tbl can be converted to a grouped variant that makes performing "by group" operations easy.:

carriers_df  <- flights %>% group_by(carrier)
carriers_db1 <- flights_db1 %>% group_by(carrier)
carriers_db2 <- flights_db2 %>% group_by(carrier)

Single table verbs

dplyr implements the following data manipulation verbs :

  • select(): selects a subset of variables
  • filter(): selects a subset of observations
  • mutate(): adds new variables
  • summarise(): reduces a group(s) to a smaller number of values (e.g., summary statistics)
  • arrange(): re-orders observations

They all work as similarly as possible across the range of data sources. The main difference is performance:

system.time(carriers_df %>% summarise(delay = mean(arr_delay)))
#>    user  system elapsed 
#>   0.036   0.001   0.037
system.time(carriers_db1 %>% summarise(delay = mean(arr_delay)) %>% collect())
#>    user  system elapsed 
#>   0.263   0.130   0.392
system.time(carriers_db2 %>% summarise(delay = mean(arr_delay)) %>% collect())
#>    user  system elapsed 
#>   0.016   0.001   0.151

Data frame methods are much much faster than their plyr equivalents. The database methods are slower, but can work with data that don't fit in memory.

system.time(plyr::ddply(flights, "carrier", plyr::summarise,
  delay = mean(arr_delay, na.rm = TRUE)))
#>    user  system elapsed 
#>   0.100   0.032   0.133

do()

As well as the specialised operations described above, dplyr also provides a generic do() function that applies any R function to each group of the data.

Let's take the batting database from the built-in Lahman database. We'll group it by year, and then fit a model to explore the relationship between the number of at bats and runs:

by_year <- lahman_df() %>% 
  tbl("Batting") %>%
  group_by(yearID)
by_year %>% 
  do(mod = lm(R ~ AB, data = .))
#> Source: local data frame [143 x 2]
#> Groups: <by row>
#> 
#>    yearID     mod
#>     (int)   (chr)
#> 1    1871 <S3:lm>
#> 2    1872 <S3:lm>
#> 3    1873 <S3:lm>
#> 4    1874 <S3:lm>
#> 5    1875 <S3:lm>
#> 6    1876 <S3:lm>
#> 7    1877 <S3:lm>
#> 8    1878 <S3:lm>
#> 9    1879 <S3:lm>
#> 10   1880 <S3:lm>
#> ..    ...     ...

Note that if you are fitting lots of linear models, it's a good idea to use biglm because it creates model objects that are considerably smaller:

by_year %>% 
  do(mod = lm(R ~ AB, data = .)) %>%
  object.size() %>%
  print(unit = "MB")
#> 22.2 Mb

by_year %>% 
  do(mod = biglm::biglm(R ~ AB, data = .)) %>%
  object.size() %>%
  print(unit = "MB")
#> 0.8 Mb

Multiple table verbs

Besides verbs that work on a single tbl, there are also a set of verbs that work with pairs of tbls: joins and set operations.

dplyr implements the four most useful SQL joins:

  • inner_join(x, y): matching x + y
  • left_join(x, y): all x + matching y
  • semi_join(x, y): all x with match in y
  • anti_join(x, y): all x without match in y

And provides methods for:

  • intersect(x, y): all rows in both x and y
  • union(x, y): rows in either x or y
  • setdiff(x, y): rows in x, but not y

Plyr compatibility

You'll need to be a little careful if you load both plyr and dplyr at the same time. I'd recommend loading plyr before dplyr, so that the faster dplyr functions come first in the search path. By and large, any function provided by both dplyr and plyr works in a similar way, but dplyr functions tend to be faster and more general.

Functions in dplyr

Name Description
as_data_frame Coerce a list to a data frame.
backend_db Database generics.
bench_compare Evaluate, compare, benchmark operations of a set of srcs.
between Do values in a numeric vector fall in specified range?
funs Create a list of functions calls.
backend_sql SQL generation.
join Join two tbls together.
progress_estimated Progress bar with estimated time.
group_size Calculate group sizes.
desc Descending order.
frame_data Row-wise data_frame creation
cumall Cumulativate versions of any, all, and mean
src_postgres Connect to postgresql.
failwith Fail with specified value.
dplyr-cluster Cluster management.
copy_to Copy a local data frame to a remote src.
arrange Arrange rows by variables.
do Do arbitrary operations on a tbl.
sql_quote Helper function for quoting sql elements.
rowwise Group input by rows
grouped_df A grouped data frame.
lead-lag Lead and lag.
tbl_cube A data cube tbl.
partial_eval Partially evaluate an expression.
grouped_dt A grouped data table.
src Create a "src" object
location Print the location in memory of a data frame
tbl_sql Create an SQL tbl (abstract)
as.tbl_cube Coerce an existing data structure into a
compute Compute a lazy tbl.
ranking Windowed rank functions.
id Compute a unique numeric id for each unique row in a data frame.
print.tbl_df Tools for describing matrices
explain Explain details of a tbl.
data_frame Build a data frame.
base_scalar Create an sql translator
filter Return rows with matching conditions.
translate_sql Translate an expression to sql.
query Create a mutable query object.
groups Get/set the grouping variables for tbl.
chain Chain together multiple operations.
distinct Select distinct/unique rows.
all.equal.tbl_df Provide a useful implementation of all.equal for data.frames.
join.tbl_sql Join sql tbls.
sql SQL escaping.
glimpse Get a glimpse of your data.
add_rownames Convert row names to an explicit variable.
n The number of observations in the current group.
build_sql Build a SQL string.
copy_to.src_sql Copy a local data frame to a sqlite src.
join.tbl_df Join data frame tbls.
src_local A local source.
slice Select rows by position.
sample Sample n rows from a table.
join.tbl_dt Join data table tbls.
temp_srcs Connect to temporary data sources.
setops Set operations.
tally Counts/tally observations by group.
same_src Figure out if two sources are the same (or two tbl have the same source)
summarise Summarise multiple values to a single value.
tbl_dt Create a data table tbl.
nycflights13 Database versions of the nycflights13 data
backend_src Source generics.
knit_print.trunc_mat knit_print method for trunc mat
dplyr dplyr: a grammar of data manipulation
mutate Add new variables.
bind Efficiently bind multiple data frames by row and column.
tbl_df Create a data frame tbl.
top_n Select top n rows (by value).
src_sql Create a "sql src" object
n_distinct Efficiently count the number of unique values in a vector.
src_mysql Connect to mysql/mariadb.
lahman Cache and retrieve an
select Select/rename variables by name.
order_by A helper function for ordering window function output.
group_by Group a tbl by one or more variables.
with_order Run a function with one order, translating result back to original order
type_sum Provide a succint summary of a type
src_sqlite Connect to a sqlite database.
group_indices Group id.
src_tbls List all tbls provided by a source.
nth Extract the first, last or nth value from a vector.
summarise_each Summarise and mutate multiple columns.
make_tbl Create a "tbl" object
tbl Create a table from a data source
select_vars Select variables.
nasa NASA spatio-temporal data
tbl_vars List variables provided by a tbl.
No Results!

Last month downloads

Details

Type Package
URL https://github.com/hadley/dplyr
BugReports https://github.com/hadley/dplyr/issues
VignetteBuilder knitr
LazyData yes
LinkingTo Rcpp (>= 0.12.0), BH (>= 1.58.0-1)
License MIT + file LICENSE
Collate 'RcppExports.R' 'all-equal.r' 'bench-compare.r' 'chain.r' 'cluster.R' 'colwise.R' 'compute-collect.r' 'copy-to.r' 'data-lahman.r' 'data-nasa.r' 'data-nycflights13.r' 'data-temp.r' 'data.r' 'dataframe.R' 'dbi-s3.r' 'desc.r' 'distinct.R' 'do.r' 'dplyr.r' 'explain.r' 'failwith.r' 'frame-data.R' 'funs.R' 'glimpse.R' 'group-by.r' 'group-indices.R' 'group-size.r' 'grouped-df.r' 'grouped-dt.r' 'id.r' 'inline.r' 'join.r' 'lead-lag.R' 'location.R' 'manip.r' 'nth-value.R' 'order-by.R' 'over.R' 'partial-eval.r' 'progress.R' 'query.r' 'rank.R' 'rbind.r' 'rowwise.r' 'sample.R' 'select-utils.R' 'select-vars.R' 'sets.r' 'sql-escape.r' 'sql-star.r' 'src-local.r' 'src-mysql.r' 'src-postgres.r' 'src-sql.r' 'src-sqlite.r' 'src.r' 'tally.R' 'tbl-cube.r' 'tbl-df.r' 'tbl-dt.r' 'tbl-sql.r' 'tbl.r' 'top-n.R' 'translate-sql-helpers.r' 'translate-sql-base.r' 'translate-sql-window.r' 'translate-sql.r' 'type-sum.r' 'utils-dt.R' 'utils-format.r' 'utils.r' 'view.r' 'zzz.r'

Include our badge in your README

[![Rdoc](http://www.rdocumentation.org/badges/version/dplyr)](http://www.rdocumentation.org/packages/dplyr)