filter_raw_data: Filter raw data locally

Description

Filter downloaded full raw dataset on local computer if the get_eurostat_data has not provided data due to too large datasets for the REST API.

Usage

filter_raw_data(raw_data = NULL, filter_table = NULL, date_filter = FALSE)

Value

a filtered data.table containing only the rows of raw_data which fulfills the conditions in the filter_table

Arguments

raw_data: an input data.table dataset resulted from the call of the get_eurostat_raw function
filter_table: a data table with values for the concepts or time to be filtered out which can be generated by the create_filter_table function
date_filter: a logical value. If TRUE the filter table should be applied to the time columns of the raw_data. The default is FALSE, in this case the filters applied to the other columns of the raw_data.

Details

It is a sub-function to use in the get_eurostat_data to filter data on the local computer if the direct response from REST API did not provide data because of too large data set (more than 30 thousands observations). The filter_table contains always at least two columns. In case if date_filter=TRUE then the two columns should have the following names and the provided conditions are applied to the time column of the the raw_data data.table.

`sd`	Starting date to be included, where date is formatted as yyyy[-mm][-dd] (the month and day are optional)
`ed`	End date of the period to be included in the dataset formatted as yyyy[-mm][-dd] (the month and day are optional)

In case if date_filter=FALSE then the columns should have the following names:

`concept`	Containing concept names, which is a column name in the `raw_data` data.table
`code`	A possible code under the given concept, which is a value in the column of the `raw_data` data.table defined by the concept

Examples

Run this code

# \dontshow{

# }
# \donttest{
id<-"tus_00age"
if (!(grepl("amzn|-aws|-azure ",Sys.info()['release']))) options(timeout=2)
rd<-get_eurostat_raw(id)
dsd<-get_eurostat_dsd(id)
ft<-create_filter_table(c("TIME_SP","Hungary",'T'),FALSE,dsd)
filter_raw_data(rd,ft)
options(timeout=60)
# }