rtemis (version 1.4.3)

ddb_data: Read CSV using DuckDB

Description

Lazy-read a CSV file, optionally: filter rows, remove duplicates, clean column names, convert character to factor, collect.

Usage

ddb_data(
  filename,
  datadir = NULL,
  sep = ",",
  header = TRUE,
  quotechar = "",
  ignore_errors = TRUE,
  make_unique = TRUE,
  select_columns = NULL,
  filter_column = NULL,
  filter_vals = NULL,
  character2factor = FALSE,
  collect = TRUE,
  progress = TRUE,
  returnobj = c("data.table", "data.frame"),
  data.table.key = NULL,
  clean_colnames = TRUE,
  verbosity = 1L
)

Value

data.frame or data.table if collect is TRUE, otherwise a character with the SQL query

Arguments

filename

Character: file name; either full path or just the file name, if datadir is also provided.

datadir

Character: Optional path if filename is not full path.

sep

Character: Field delimiter/separator.

header

Logical: If TRUE, first line will be read as column names.

quotechar

Character: Quote character.

ignore_errors

Logical: If TRUE, ignore parsing errors.

make_unique

Logical: If TRUE, keep only unique rows.

select_columns

Character vector: Column names to select.

filter_column

Character: Name of column to filter on, e.g. "ID".

filter_vals

Numeric or Character vector: Values in filter_column to keep.

character2factor

Logical: If TRUE, convert character columns to factors.

collect

Logical: If TRUE, collect data and return structure class as defined by returnobj.

progress

Logical: If TRUE, enable the DuckDB progress bar.

returnobj

Character: "data.frame" or "data.table" object class to return. If "data.table", data.frame object returned from DBI::dbGetQuery is passed to data.table::setDT.

data.table.key

Character: If set, this corresponds to a column name in the dataset. This column will be set as key in the data.table output.

clean_colnames

Logical: If TRUE, clean colnames with clean_colnames.

verbosity

Integer: Verbosity level.

Author

EDG

Examples

if (FALSE) {
# Requires local CSV file; replace with your own path
ir <- ddb_data("/Data/massive_dataset.csv",
  filter_column = "ID",
  filter_vals = 8001:9999
)
}