neonstore (version 0.4.4)

neon_download_s3: Download requested NEON files from an S3 bucket

Description

It is possible to copy the local neonstore (see [neon_dir()]) to an S3 bucket for faster shared access to a large store. This function mimics the behavior of [neon_download()] but accesses files directly from such an S3 bucket. Queries the AWS-S3 REST endpoint GET bucket for a file list in (in 1000-file chunks), then filters file names to determine what to download. Users must set the api to the address of their S3 bucket to take advantage of this feature. For demonstration purposes, an example S3 bucket is provided as the default. Note that data obtained in this way is only as up-to-date or complete as the underlying cache. Users should always draw from the NEON API using [neon_download()] to ensure they have the most recent and complete data files.

Usage

neon_download_s3(
  product,
  table = NA,
  site = NA,
  start_date = NA,
  end_date = NA,
  type = "basic",
  release = NA,
  quiet = FALSE,
  verify = TRUE,
  dir = neon_dir(),
  get_zip = FALSE,
  unzip = FALSE,
  api = "https://minio.thelio.carlboettiger.info/neonstore/"
)

Value

(invisibly) table of requested files and metadata

Arguments

product

A NEON productCode or list of product codes, see examples.

table

Include only files matching this table name (or regex pattern). (optional).

site

4-letter site code(s) to filter on. Leave as NA to search all.

start_date

Download only files as recent as (YYYY-MM-DD). Leave as NA to download up to the most recent available data.

end_date

Download only files up to end_date (YYYY-MM-DD). Leave as NA to download all prior data.

type

Should we prefer the basic or expanded version of this product? Note that not all products have expanded formats.

release

Select only data files associated with a particular release tag, see https://www.neonscience.org/data-samples/data-management/data-revisions-releases, e.g. "RELEASE-2021". Releases are associated with a specific DOI and the promise that files associated with a particular release will not change.

quiet

Should download progress be displayed?

verify

Should downloaded files be compared against the MD5 hash reported by the NEON API to verify integrity? (default TRUE)

dir

Location where files should be downloaded. By default will use the appropriate applications directory for your system (see tools::R_user_dir()). This default also be configured by setting the environmental variable NEONSTORE_HOME, see Sys.setenv or Renviron.

get_zip

should we attempt to download .zip archive versions of files? default FALSE, as zip archives are being deprecated from NEON API starting in early 2021.

unzip

should we extract .zip files? (default TRUE). Note: .zip files are preserved in the store to avoid repeated downloads. Use of .zip files in NEON API is now deprecated in favor of requesting individual files.

api

URL to an S3 bucket containing raw NEON data files (in flat file structure like that used by neonstore).

Details

Note: at this time, release information associated with files in the store is not available from neonstore S3 caches. As such, this mechanism is not able to filter data for specific RELEASE tags, is which are only available from the NEON API. Querying products by [neon_download()] will update the corresponding release tags.

Examples

# \dontshow{
# Users get confused by use of tempdir in examples, so hide
Sys.setenv(NEONSTORE_HOME=tempdir())
# }

# \donttest{
 neon_download("DP1.10003.001", 
               start_date = "2018-01-01", 
               end_date = "2019-01-01",
               site = "YELL")
# }

# \dontshow{
 # And clean up
Sys.unsetenv("NEONSTORE_HOME")
# }