Create a list of all the files (in all subfolders) of an FTP server.
Defaults to the German Weather Service (DWD, Deutscher WetterDienst) OpenData server at
ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/.
The R package RCurl
must be available to do this.
If RCurl::getURL
fails, usually because bot access is
detected and denied, there will still be an output which you can pass in a
second run via folder
to extract the remaining dirs.
You might want to wait a bit and set sleep
to a higher value in that case.
Here's an example:
gridindex <- indexFTP("", gridbase)
gridindex <- indexFTP(gridindex, gridbase, sleep=1)
indexFTP(folder = "currentfindex", base = dwdbase,
is.file.if.has.dot = TRUE, sleep = 0, dir = "DWDdata",
filename = folder[1], overwrite = FALSE, quiet = FALSE,
progbar = !quiet, verbose = FALSE)
Folder(s) to be indexed recursively, e.g. "/hourly/wind/".
Leading slashes will be removed.
Use folder=""
to search at the location of base
itself.
If folder
is "currentfindex" (the default) and base
is the default, folder
is changed to all observational
folders listed in the current tree file at
ftp://opendata.dwd.de/weather/tree.html. With "currentgindex"
and gridbase
, the grid folders in the tree are used.
DEFAULT: "currentfindex"
Main directory of FTP server. Trailing slashes will be removed.
DEFAULT: dwdbase
Logical: if some of the input paths contain a dot, treat those as files, i.e. do not try to read those as if they were a folder. Only set this to FALSE if you know what you're doing. DEFAULT: TRUE
If not 0, a random number of seconds between 0 and sleep
is passed to Sys.sleep
after each read folder
to avoid getting kicked off the FTP-Server. DEFAULT: 0
Writeable directory name where to save the downloaded file.
Created if not existent.
DEFAULT: "DWDdata" at current getwd()
Character: Part of output filename. "INDEX_of_DWD_" is prepended, "/" replaced with "_", ".txt" appended. DEFAULT: folder[1]
Logical: Overwrite existing file? If not, "_n" is added to the
filename, see berryFunctions::newFilename
.
DEFAULT: FALSE
Suppress progbars and message about directory/files? DEFAULT: FALSE
Logical: present a progress bar in each level? DEFAULT: TRUE
Logical: write a lot of messages from RCurl::getURL
?
DEFAULT: FALSE (usually, you dont need all the curl information)
a vector with file paths
It's not suggested to run this for all folders, as it can take quite some time
and you may get kicked off the FTP-Server. This package contains an index
of the climatic observations at weather stations: View(rdwd:::fileIndex)
.
If it is out of date, please let me know!
# NOT RUN {
## Needs internet connection
sol <- indexFTP(folder="/daily/solar", dir=tempdir())
head(sol)
# mon <- indexFTP(folder="/monthly/kl", dir=tempdir(), verbose=TRUE)
# }
# NOT RUN {
# }
Run the code above in your browser using DataLab