ifiles
From text2vec v0.6
by Dmitriy Selivanov
Creates iterator over text files from the disk
The result of this function usually used in an itoken function.
Usage
ifiles(file_paths, reader = readLines)idir(path, reader = readLines)
ifiles_parallel(file_paths, reader = readLines, ...)
Arguments
- file_paths
characterpaths of input files- reader
functionwhich will perform reading of text files from disk, which should take a path as its first argument.reader()function should return named character vector: elements of vector = documents, names of the elements = document ids which will be used in DTM construction. If user doesn't provide named character vector, document ids will be generated as file_name + line_number (assuming that each line is a document).- path
characterpath of directory. All files in the directory will be read.- ...
other arguments (not used at the moment)
See Also
Examples
# NOT RUN {
current_dir_files = list.files(path = ".", full.names = TRUE)
files_iterator = ifiles(current_dir_files)
parallel_files_iterator = ifiles_parallel(current_dir_files, n_chunks = 4)
it = itoken_parallel(parallel_files_iterator)
dtm = create_dtm(it, hash_vectorizer(2**16), type = 'dgTMatrix')
# }
# NOT RUN {
dir_files_iterator = idir(path = ".")
# }
Community examples
Looks like there are no examples yet.