textfile: read a text corpus source from a file

Description

Read a text corpus from a source file, where the single file will consist of a set of texts in columns and document variables and document-level meta-data in additional columns. For spreadsheet-like files, the first row must be a header.

Usage

textfile(file, textField, cache = FALSE, docvarsfrom = c("filenames"),
  dvsep = "_", docvarnames = NULL, ...)
## S3 method for class 'character,index,ANY,missing,missing,missing':
textfile(file,
  textField, cache = FALSE, docvarsfrom = c("filenames"), dvsep = "_",
  docvarnames = NULL, ...)
## S3 method for class 'character,missing,ANY,missing,missing,missing':
textfile(file,
  textField, cache = FALSE, docvarsfrom = c("filenames"), dvsep = "_",
  docvarnames = NULL, ...)
## S3 method for class 'character,missing,ANY,character,ANY,ANY':
textfile(file,
  textField = NULL, cache = FALSE, docvarsfrom = c("headers"),
  dvsep = "_", docvarnames = NULL, ...)

Arguments

file

the complete filename(s) to be read. The value can be a vector of file names, a single file name, or a file "mask" using a "glob"-type wildcard value. Currently available file value types are: [object Object],[object Object],[object Object],[object Ob

textField

a variable (column) name or column number indicating where to find the texts that form the documents for the corpus. This must be specified for file types .csv and .json.

cache

If TRUE, write the object to a temporary file and store the temporary filename in the corpusSource-class object definition. If FALSE, return the data in the object. Caching the fil

docvarsfrom

used to specify that docvars should be taken from the filenames, when the textfile inputs are filenames and the elements of the filenames are document variables, separated by a delimiter (dvsep). This allows easy assignment o

dvsep

separator used in filenames to delimit docvar elements if docvarsfrom="filenames" is used

docvarnames

character vector of variable names for docvars, if docvarsfrom is specified. If this argument is not used, default docvar names will be used (docvar1, docvar2, ...).

...

additional arguments passed through to low-level file reading function, such as file, read.csv, etc. Useful for specifying an input encoding option, which i

Value

an object of class corpusSource-class that can be read by corpus to construct a corpus

Details

The constructor does not store a copy of the texts, but rather reads in the texts and associated data, and saves them to a temporary disk file whose location is specified in the corpusSource-class object. This prevents a complete copy of the object from cluttering the global environment and consuming additional space. This does mean however that the state of the file containing the source data will not be cross-platform and may not be persistent across sessions. So the recommended usage is to load the data into a corpus in the same session in which textfile is called.

Examples

Run this code

# Twitter json
mytf1 <- textfile("http://www.kenbenoit.net/files/tweets.json")
summary(corpus(mytf1), 5)
# generic json - needs a textField specifier
mytf2 <- textfile("http://www.kenbenoit.net/files/sotu.json",
                  textField = "text")
summary(corpus(mytf2))
# text file
mytf3 <- textfile(unzip(system.file("extdata", "pg2701.txt.zip", package = "quanteda")))
summary(corpus(mytf3))
# XML data
mytf6 <- textfile("http://www.kenbenoit.net/files/plant_catalog.xml", 
                  textField = "COMMON")
summary(corpus(mytf6))
# csv file
write.csv(data.frame(inaugSpeech = texts(inaugCorpus), docvars(inaugCorpus)), 
          file = "/tmp/inaugTexts.csv", row.names = FALSE)
mytf7 <- textfile("/tmp/inaugTexts.csv", textField = "inaugSpeech")
summary(corpus(mytf7))

# vector of full filenames for a recursive structure
textfile(list.files(path = "~/Desktop/texts", pattern = "\\.txt$", 
                    full.names = TRUE, recursive = TRUE))

Run the code above in your browser using DataLab