indexFTP: Create a recursive index of an FTP Server

Description

Create a list of all the files (in all subfolders) of an FTP server. Defaults to the German Weather Service (DWD, Deutscher WetterDienst) OpenData server at ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/. The R package RCurl must be available to do this. If RCurl::getURL fails, usually because bot access is detected and denied, there will still be an output which you can pass in a second run via folder to extract the remaining dirs. You might want to wait a bit and set sleep to a higher value in that case. Here's an example: gridindex <- indexFTP("", gridbase) gridindex <- indexFTP(gridindex, gridbase, sleep=1)

Usage

indexFTP(folder = "currentfindex", base = dwdbase,
  is.file.if.has.dot = TRUE, sleep = 0, dir = "DWDdata",
  filename = folder[1], overwrite = FALSE, quiet = FALSE,
  progbar = !quiet, verbose = FALSE)

Arguments

folder

Folder(s) to be indexed recursively, e.g. "/hourly/wind/". Leading slashes will be removed. Use folder="" to search at the location of base itself. If folder is "currentfindex" (the default) and base is the default, folder is changed to all observational folders listed in the current tree file at ftp://opendata.dwd.de/weather/tree.html. With "currentgindex" and gridbase, the grid folders in the tree are used. DEFAULT: "currentfindex"

base

Main directory of FTP server. Trailing slashes will be removed. DEFAULT: dwdbase

is.file.if.has.dot

Logical: if some of the input paths contain a dot, treat those as files, i.e. do not try to read those as if they were a folder. Only set this to FALSE if you know what you're doing. DEFAULT: TRUE

sleep

If not 0, a random number of seconds between 0 and sleep is passed to Sys.sleep after each read folder to avoid getting kicked off the FTP-Server. DEFAULT: 0

dir

Writeable directory name where to save the downloaded file. Created if not existent. DEFAULT: "DWDdata" at current getwd()

filename

Character: Part of output filename. "INDEX_of_DWD_" is prepended, "/" replaced with "_", ".txt" appended. DEFAULT: folder[1]

overwrite

Logical: Overwrite existing file? If not, "_n" is added to the filename, see berryFunctions::newFilename. DEFAULT: FALSE

quiet

Suppress progbars and message about directory/files? DEFAULT: FALSE

progbar

Logical: present a progress bar in each level? DEFAULT: TRUE

verbose

Logical: write a lot of messages from RCurl::getURL? DEFAULT: FALSE (usually, you dont need all the curl information)

Value

a vector with file paths

Details

It's not suggested to run this for all folders, as it can take quite some time and you may get kicked off the FTP-Server. This package contains an index of the climatic observations at weather stations: View(rdwd:::fileIndex). If it is out of date, please let me know!

Examples

Run this code

# NOT RUN {
 ## Needs internet connection
sol <- indexFTP(folder="/daily/solar", dir=tempdir())
head(sol)

# mon <- indexFTP(folder="/monthly/kl", dir=tempdir(), verbose=TRUE)
# }
# NOT RUN {
# }