get_eurostat_data: Download/extract Eurostat Data

Description

Download full or partial data set from Eurostat database.

Usage

get_eurostat_data(id, filters = NULL, exact_match = TRUE,
  date_filter = NULL, label = FALSE, select_freq = NULL,
  cache = TRUE, update_cache = FALSE, cache_dir = NULL,
  compress_file = TRUE, stringsAsFactors = default.stringsAsFactors(),
  keep_flags = FALSE, cflags = FALSE, check_toc = FALSE,
  verbose = FALSE, ...)

Arguments

A code name for the dataset of interest. See search_eurostat_toc for details how to get an id.

filters

a string or a character vector containing words to filter by the different concepts or geographical location. If filter applied only part of the dataset is downloaded through the API. The words can be any words, Eurostat variable codes, and values available in the DSD search_eurostat_dsd. The default is NULL, in this case the whole dataset is returned via the bulk download. To filter by time see date_filter below. If after filtering still the dataset has more observations than the limit per query via the API, then the bulk download is used to retrieve the data.

exact_match

a boolean with the default value TRUE, if the strings provided in filters shall be matched exactly as it is or as a pattern.

date_filter

a vector which can be numeric or character containing dates to filter the dataset. If date filter applied only part of the dataset is downloaded through the API. The default is NULL, in this case the whole dataset is returned via the bulk download. If after filtering still the dataset has more observations than the limit per query via the API, then the bulk download is used to retrieve the data.

label

a boolean with the default FALSE. If it is TRUE then the code values are replaced by the name from the Data Structure Definition (DSD) get_eurostat_dsd. For example instead of "D1110A", "Raw cows' milk from farmtype" is used or "HU32" is replaced by "<U+00C9>szak-Alf<U+00F6>ld".

select_freq

a character symbol for a time frequency when a dataset has multiple time frequencies. Possible values are: A = annual, S = semi-annual, H = half-year, Q = quarterly, M = monthly, W = weekly, D = daily. The default is NULL as most datasets have just one time frequency and in this case if there are multiple frequencies, then only the most common frequency kept. If all the frequencies needed the get_eurostat_raw can be used.

cache

a logical whether to do caching. Default is TRUE. Affects only queries without filtering. If filters or date_filter is used then there is no caching.

update_cache

a logical with a default value FALSE, whether to update the data in the cache. Can be set also with options(restatapi_update=TRUE)

cache_dir

a path to a cache directory. The NULL (default) uses the memory as cache. If the folder cache_dir directory does not exist it saves in the 'restatapi' directory under the temporary directory from tempdir(). Directory can also be set with option(restatapi_cache_dir=...).

compress_file

a logical whether to compress the RDS-file in caching. Default is TRUE.

stringsAsFactors

if TRUE (the default) the non-numeric columns are converted to factors. If the value FALSE they are returned as a characters.

keep_flags

a logical whether the observation status (flags) - e.g. "confidential", "provisional", etc. - should be kept in a separate column or if they can be removed. Default is FALSE. For flag values see: http://ec.europa.eu/eurostat/data/database/information.

cflags

a logical whether the missing observations with flag 'c' - "confidential" should be kept or not. Default is FALSE, in this case these observations dropped from the dataset. If this parameter TRUE then the flags are kept and the parameter provided in keep_flags is not taken into account.

check_toc

a boolean whether to check the provided id in the Table of Contents (TOC) or not. The default value FALSE, in this case the base URL for the download link is retrieved from the configuration file. If the value is TRUE then the TOC is downloaded and the id is checked in it. If it found then the download link is retrieved form the TOC.

verbose

A boolean with default FALSE, so detailed messages (for debugging) will not printed. Can be set also with options(restatapi_verbose=TRUE)

...

further arguments to the for search_eurostat_dsd function, e.g.: ignore.case or name. If the ignore.case has the default value FALSE, then the strings provided in filters are matched as is, otherwise the case of the letters is ignored. If the name=FALSE then the pattern(s) provided in the filters argument is only searched in the code column of the DSD, and the names of the codes will not be searched.

Value

a data.table.a data.table with the following columns: #'

`freq`	A column for the frequency of the data in case there are multiple frequencies, for single frequency this columns is dropped from the data table
dimension names	One column for each dimension in the data
`time`	A column for the time dimension
`values`	A column for numerical values

The data.table does not include all missing values. The missing values are dropped if the value and flag are missing on a particular time.

Details

Data sets are downloaded from the Eurostat Web Services SDMX API if there is a filter otherwise the the Eurostat bulk download facility is used. If only the table id is given, the whole table is downloaded from the bulk download facility. If also filters or date_filter is defined then the SDMX REST API is used. In case after filtering the dataset has more rows than the limitation of the SDMX REST API (1 million values at one time) then the bulk download is used to retrieve the whole dataset .

By default all datasets cached as they are often rather large. The datasets cached in memory (default) or can be stored in a temporary directory if cache_dir or option(restatpi_cache_dir) is defined. The cache can be emptied with clean_restatapi_cache.

The id, is a value from the code column of the table of contents (get_eurostat_toc), and can be searched for with the search_eurostat_toc function. The id value can be retrieved from the Eurostat database as well. The Eurostat database gives codes in the Data Navigation Tree after every dataset in parenthesis.

Filtering can be done by the codes as described in the API documentation providing in the correct order and connecting with "." and "+". If we do not know the codes we can filter based on words or by the mix of the two putting in a vector like c("AT$","Belgium","persons","Total"). Be careful that the filter is case sensitive, if you do not know the code or label exactly you can use the option ignore.case=TRUE and exact_match=FALSE, but in this case the results may include unwanted elements as well. In the filters parameter regular expressions can be used as well. We do not have to worry about the correct order of the filter, it will be put in the correct place based on the DSD.

The date_filter shall be a string in the format yyyy[-mm][-dd]. The month and the day part is optional, but if we use the years and we have monthly frequency then all the data for the given year is retrieved. The string can be extended by adding the "<" or ">" to the beginning or to the end of the string. In this case the date filter is treated as range, and the date is used as a starting or end date. The data will include the observation of the start/end date. A single date range can be defined as well by concatenating two dates with the ":", e.g. "2016-08:2017-03-15". As seen in the example the dates can have different length: one defined only at year/month level, the other by day level. If a date range is defined with ":", it is not possible to use the "<" or ">" characters in the date filter. If there are multiple dates which is not a continuous range, it can be put in vector in any order like c("2016-08",2013:2015,"2017-07-01"). In this case, as well, it is not possible to use the "<" or ">" characters.

Examples

Run this code

# NOT RUN {
load_cfg()
eu<-get("cc",envir=.restatapi_env)

# }
# NOT RUN {
dt<-get_eurostat_data("NAMA_10_GDP")
dt<-get_eurostat_data("nama_10_gdp",update_cache=TRUE)
dt<-get_eurostat_data("nama_10_gdp",cache_dir="/tmp")
options(restatapi_update=FALSE)
options(restatapi_cache_dir=file.path(tempdir(),"restatapi"))
dt<-get_eurostat_data("avia_gonc",select_freq="A",cache=FALSE)
dt<-get_eurostat_data("agr_r_milkpr",date_filter=2008,keep_flags=TRUE)
dt<-get_eurostat_data("avia_par_me",
                      filters="BE$",
                      exact_match=FALSE,
                      date_filter=c(2016,"2017-03","2017-07-01"),
                      select_freq="Q",
                      label=TRUE,
                      name=FALSE)
dt<-get_eurostat_data("agr_r_milkpr",
                      filters=c("BE$","Hungary"),
                      date_filter="2007-06<",
                      keep_flags=TRUE)
dt<-get_eurostat_data("nama_10_a10_e",
                      filters=c("Annual","EU28","Belgium","AT","Total","EMP_DC","person"),
                      date_filter=c("2008",2002,2013:2018))
dt<-get_eurostat_data("vit_t3",
                      filters=c("EU28",eu$EA15,"HU$"),
                      date_filter=c("2015",2007))
dt<-get_eurostat_data("avia_par_me",
                      filters="Q...ME_LYPG_HU_LHBP+ME_LYTV_UA_UKKK",
                      date_filter=c("2016-08","2017-07-01"),
                      select_freq="M")
# }