rtemis (version 1.4.3)

ingest: Read a data file and normalize it to Parquet

Description

Reads a data file with the settings of an IngestConfig, converts the columns it declares types for, writes the result to a Parquet file, and returns a manifest of the operation.

The file extension names the format. A config whose format differs from it is an error.

path may be an http:// or https:// URL or an hf://datasets/ path, as read() accepts them, downloaded to a temporary file. The manifest records the path as given in input, the time the download started in retrieved_at, and, for an hf:// path, the path pinned to the resolved commit and file in input_resolved.

The manifest also records the input and output files, the SHA-256 of the input, a DataFingerprint of the data, the config, and the engine that produced the Parquet file.

Usage

ingest(path, outfile, config = NULL, overwrite = FALSE, verbosity = 1L)

Value

Named list: the ingest manifest, with input, input_resolved

(hf://datasets/<repo>@<commit>/<file> for an hf:// input, otherwise NULL), input_sha256 (SHA-256 of the input file's bytes), retrieved_at

(ISO 8601 UTC time the download started, or NULL for a local file), output, format, config, engine, rtemis_version, data_input, n_rows, and n_cols.

Arguments

path

Character: Path to the input file, an http:// or https:// URL, or an hf://datasets/ path.

outfile

Character: Path to write the Parquet file to.

config

Optional IngestConfig: How to read it. If NULL, the defaults for the file's own format.

overwrite

Logical: If TRUE, overwrite outfile if it exists.

verbosity

Integer: Verbosity level.

Author

EDG

Examples

if (FALSE) {
ingest("data.csv", "data.parquet")
ingest("data.csv", "data.parquet", setup_DelimitedIngest(sep = ";"))
ingest("https://example.org/data/iris.csv", "iris.parquet")
ingest("hf://datasets/rtemis/tcga-pancan-hiseq", "tcga.parquet")
}