spanishoddata (version 0.1.0)

spod_duckdb_limit_resources: Set maximum memory and number of threads for a DuckDB connection

Description

Set maximum memory and number of threads for a DuckDB connection

Usage

spod_duckdb_limit_resources(
  con,
  max_mem_gb = max(4, spod_available_ram() - 4),
  max_n_cpu = parallelly::availableCores() - 1
)

Value

A duckdb connection.

Arguments

con

A duckdb connection

max_mem_gb

The maximum memory to use in GB. A conservative default is 3 GB, which should be enough for resaving the data to DuckDB form a folder of CSV.gz files while being small enough to fit in memory of most even old computers. For data analysis using the already converted data (in DuckDB or Parquet format) or with the raw CSV.gz data, it is recommended to increase it according to available resources.

max_n_cpu

The maximum number of threads to use. Defaults to the number of available cores minus 1.