Arrow Datasets allow you to query against data that has been split across
multiple files. This sharding of data may indicate partitioning, which
can accelerate queries that only touch some partitions (files). Call
open_dataset() to point to a directory of data files and return a
Dataset, then use dplyr methods to query it.
open_dataset(
sources,
schema = NULL,
partitioning = hive_partition(),
unify_schemas = NULL,
...
)One of:
a string path or URI to a directory containing data files
a string path or URI to a single file
a character vector of paths or URIs to individual data files
a list of Dataset objects as created by this function
a list of DatasetFactory objects as created by dataset_factory().
When sources is a vector of file URIs, they must all use the same protocol
and point to files located in the same file system and having the same
format.
Schema for the Dataset. If NULL (the default), the schema
will be inferred from the data sources.
When sources is a directory path/URI, one of:
a Schema, in which case the file paths relative to sources will be
parsed, and path segments will be matched with the schema fields. For
example, schema(year = int16(), month = int8()) would create partitions
for file paths like "2019/01/file.parquet", "2019/02/file.parquet",
etc.
a character vector that defines the field names corresponding to those
path segments (that is, you're providing the names that would correspond
to a Schema but the types will be autodetected)
a HivePartitioning or HivePartitioningFactory, as returned
by hive_partition() which parses explicit or autodetected fields from
Hive-style path segments
NULL for no partitioning
The default is to autodetect Hive-style partitions. When sources is not a
directory path/URI, partitioning is ignored.
logical: should all data fragments (files, Datasets)
be scanned in order to create a unified schema from them? If FALSE, only
the first fragment will be inspected for its schema. Use this fast path
when you know and trust that all fragments have an identical schema.
The default is FALSE when creating a dataset from a directory path/URI or
vector of file paths/URIs (because there may be many files and scanning may
be slow) but TRUE when sources is a list of Datasets (because there
should be few Datasets in the list and their Schemas are already in
memory).
additional arguments passed to dataset_factory() when sources
is a directory path/URI or vector of file paths/URIs, otherwise ignored.
These may include format to indicate the file format, or other
format-specific options.
A Dataset R6 object. Use dplyr methods on it to query the data,
or call $NewScan() to construct a query directly.
vignette("dataset", package = "arrow")