Source

SourceFactory

FileSystemSourceFactory

A <a rd-options="" href="/link/Dataset?package=arrow&version=0.16.0" data-mini-rdoc="arrow::Dataset">Dataset</a> can have one or more <code>Source</code>s. A <code>Source</code> contains one or more
<code>Fragments</code>, such as files, of a common type and partitioning.
<code>SourceFactory</code> is used to create a <code>Source</code>, inspect the <a rd-options="" href="/link/Schema?package=arrow&version=0.16.0" data-mini-rdoc="arrow::Schema">Schema</a> of the
fragments contained in it, and declare a partitioning.
<code>FileSystemSourceFactory</code> is a subclass of <code>SourceFactory</code> for
discovering files in the local file system, the only currently supported
file system.
In general, you'll deal with <code>SourceFactory</code> rather than <code>Source</code> itself.
Return the Source's <code>Schema</code>

'Apache' 'Arrow' <https://arrow.apache.org/> is a cross-language
development platform for in-memory data. It specifies a standardized
language-independent columnar memory format for flat and hierarchical data,
organized for efficient analytic operations on modern hardware. This
package provides an interface to the 'Arrow C++' library.

Neal Richardson

arrow

Integration to 'Apache' 'Arrow'

Romain François

Jeroen Ooms

Javier Luraschi

Jeffrey Wong

Apache Arrow 

Source function

For the <code>SourceFactory$create()</code> factory method, see <code><a rd-options="=open_source" href="/link/open_source()?package=arrow&version=0.16.0&to=%3Dopen_source" data-mini-rdoc="=open_source::open_source()">open_source()</a></code>, an
alias for it.<code>FileSystemSourceFactory$create()</code> is a lower-level factory method and
takes the following arguments:<ul>
<li><code>filesystem</code>: A <a rd-options="" href="/link/FileSystem?package=arrow&version=0.16.0" data-mini-rdoc="arrow::FileSystem">FileSystem</a></li>
<li><code>selector</code>: A <a rd-options="" href="/link/FileSelector?package=arrow&version=0.16.0" data-mini-rdoc="arrow::FileSelector">FileSelector</a></li>
<li><code>format</code>: A string identifier of the format of the files in <code>path</code>.
Currently supported options are "parquet", "arrow", and "ipc" (an alias for
the Arrow file format)</li>
</ul>

Factory

<code>Source</code> has one defined method:<ul>
<li><code>$schema</code>: Active binding, returns the <a rd-options="" href="/link/Schema?package=arrow&version=0.16.0" data-mini-rdoc="arrow::Schema">Schema</a> of the <code>Source</code></li>
</ul><code>SourceFactory</code> and its subclasses have the following methods:<ul>
<li><code>$Inspect()</code>: Walks the files in the directory and returns a common <a rd-options="" href="/link/Schema?package=arrow&version=0.16.0" data-mini-rdoc="arrow::Schema">Schema</a></li>
<li><code>$Finish(schema)</code>: Returns a <code>Source</code></li>
</ul>

Methods

A <a rd-options='' href='Dataset'>Dataset</a> can have one or more <code>Source</code>s. A <code>Source</code> contains one or more
<code>Fragments</code>, such as files, of a common type and partitioning.
<code>SourceFactory</code> is used to create a <code>Source</code>, inspect the <a rd-options='' href='Schema'>Schema</a> of the
fragments contained in it, and declare a partitioning.
<code>FileSystemSourceFactory</code> is a subclass of <code>SourceFactory</code> for
discovering files in the local file system, the only currently supported
file system.
In general, you'll deal with <code>SourceFactory</code> rather than <code>Source</code> itself.
Return the Source's <code>Schema</code>

Sources for a Dataset — Source

<code>Source</code> has one defined method:<ul>
<li><code>$schema</code>: Active binding, returns the <a rd-options='' href='Schema'>Schema</a> of the <code>Source</code></li>
</ul><code>SourceFactory</code> and its subclasses have the following methods:<ul>
<li><code>$Inspect()</code>: Walks the files in the directory and returns a common <a rd-options='' href='Schema'>Schema</a></li>
<li><code>$Finish(schema)</code>: Returns a <code>Source</code></li>
</ul>

Source: Sources for a Dataset

Description

Arguments

Factory

Methods

See Also