neonstore (version 0.2.1)

neon_download: Download NEON data products into a local store

Description

Download NEON data products into a local store

Usage

neon_download(
  product,
  start_date = NA,
  end_date = NA,
  site = NA,
  type = "expanded",
  file_regex = "[.]zip",
  quiet = FALSE,
  verify = TRUE,
  dir = neon_dir(),
  unzip = TRUE,
  api = "https://data.neonscience.org/api/v0",
  .token = Sys.getenv("NEON_TOKEN")
)

Arguments

product

A NEON productCode. See neon_download.

start_date

Download only files as recent as (YYYY-MM-DD). Leave as NA to download up to the most recent available data.

end_date

Download only files up to end_date (YYYY-MM-DD). Leave as NA to download all prior data.

site

4-letter site code(s) to filter on. Leave as NA to search all.

type

Should we prefer the basic or expanded version of this product? See details.

file_regex

Download only files matching this pattern. See details.

quiet

Should download progress be displayed?

verify

Should downloaded files be compared against the MD5 hash reported by the NEON API to verify integrity? (default TRUE)

dir

Location where files should be downloaded. By default will use the appropriate applications directory for your system (see rappdirs::user_data_dir). This default also be configured by setting the environmental variable NEONSTORE_HOME, see Sys.setenv or Renviron.

unzip

should we extract .zip files? Also removes the .zip files. (default TRUE). Set to FALSE if you want to keep .zip archives and manually unzip them later.

api

the URL to the NEON API, leave as default.

.token

an authentication token from NEON. A token is not required but will allow access to a higher number of requests before rate limiting applies, see https://data.neonscience.org/data-api/rate-limiting/#api-tokens. Note that once files are downloaded once, neonstore provides persistent access to them without further interaction required with the API.

Details

Each NEON data product consists of a collection of objects (e.g. tables), which are in turn broken into individual files by site and sampling month. Additionally, many NEON products have been expanded, including some additional columns. Consequently, users must specify if they want the "basic" or "expanded" version of this data.

In the products table (see neon_products), the productHasExpanded column indicates if the data product has expanded, and the columns productHasBasicDescription and productHasExpandedDescription provide a detailed explanation of the differences between the "expanded" and "basic" versions of that particular product.

The API provides access to a .zip file containing all the component objects (e.g. tables) for that product at that site and sampling month. Additionally, the API allows users to request component files directly (e.g. as .csv files). Requesting component files directly avoids the additional overhead of downloading other components that are not needed. Both the .zip and relevant .csv and zip files in products that have expanded will include both a "basic" and "expanded" name in the filename. Setting type argument of neon_download() to the preferred one will make it filter out the other one.

By default, neon_download() will request the .zip packet for the product, matching the requested type. neon_download() will extract the component files into the store, removing the .zip file. Specific files within a product can be identified by altering the file_regex argument (see examples).

neon_download() will avoid downloading metadata files which are bitwise identical to other files in the same download request, as indicated by the crc32 hash reported by the API. These typically include metadata that are shared across the product as a whole, but are for some reason included in each sampling month for each site -- potentially thousands of duplicates. These duplicates are also packaged within the .zip downloads where it is not possible to exclude them from the download.

Examples

# NOT RUN {
 
 neon_download("DP1.10003.001", 
               start_date = "2018-01-01", 
               end_date = "2019-01-01",
               site = "YELL")
               
 ## Advanced use: filter for a particular table in the product
 neon_download(product = "DP1.10003.001",
               start_date = "2018-01-01",
               end_date = "2019-01-01",
               site = "YELL",
               file_regex = ".*brd_countdata.*\\.csv")

# }