itoken

0th

Percentile

Iterators over input objects

This function creates iterators over input objects to vocabularies, corpora, or DTM and TCM matrices. This iterator is usually used in following functions : create_vocabulary, create_corpus, create_dtm, vectorizers, create_tcm. See them for details.

Usage
itoken(iterable, ...)
"itoken"(iterable, chunks_number = 10, progessbar = interactive(), ids = NULL, ...)
"itoken"(iterable, preprocess_function = identity, tokenizer = function(x) strsplit(x, " ", TRUE), chunks_number = 10, progessbar = interactive(), ids = NULL, ...)
"itoken"(iterable, preprocess_function = identity, tokenizer = function(x) strsplit(x, " ", TRUE), progessbar = interactive(), ...)
"itoken"(iterable, preprocess_function = identity, tokenizer = function(x) strsplit(x, " ", TRUE), ...)
Arguments
iterable
an object from which to generate an iterator
...
arguments passed to other methods (not used at the moment)
chunks_number
integer, the number of pieces that object should be divided into.
progessbar
logical indicates whether to show progress bar.
ids
vector of document ids. If ids is not provided, names(iterable) will be used. If names(iterable) == NULL, incremental ids will be assigned.
preprocess_function
function which takes chunk of character vectors and does all pre-processing. Usually preprocess_function should return a character vector of preprocessed/cleaned documents. See "Details" section.
tokenizer
function which takes a character vector from preprocess_function, split it into tokens and returns a list of character vectors. If you need to perform stemming - call stemmer inside tokenizer. See examples section.
Details

S3 methods for creating an itoken iterator from list of tokens

  • list: all elements of the input list should be character vectors containing tokens
  • character: raw text source: the user must provide a tokenizer function
  • ifiles: from files, a user must provide a function to read in the file (to ifiles) and a function to tokenize it (to itoken)
  • idir: from a directory, the user must provide a function to read in the files (to idir) and a function to tokenize it (to itoken)
  • ilines: from lines, the user must provide functions to tokenize

See Also

ifiles, idir, ilines, create_vocabulary, create_corpus, create_dtm, vectorizers, create_tcm

Aliases
  • itoken
  • itoken.character
  • itoken.ifiles
  • itoken.ilines
  • itoken.list
Examples
data("movie_review")
txt <- movie_review$review[1:100]
ids <- movie_review$id[1:100]
it <- itoken(txt, tolower, word_tokenizer, chunks_number = 10)
it <- itoken(txt, tolower, word_tokenizer, chunks_number = 10, ids = ids)
# Example of stemming tokenizer
# stem_tokenizer <- function(x) {
#  word_tokenizer(x) %>% lapply(SnowballC::wordStem('en'))
# }
Documentation reproduced from package text2vec, version 0.3.0, License: MIT + file LICENSE

Community examples

xueliu.stonybrook@gmail.com at Jun 25, 2017 text2vec v0.4.0

data("movie_review") txt = movie_review$review[1:100] ids = movie_review$id[1:100] it = itoken(txt, tolower, word_tokenizer, chunks_number = 10) it = itoken(txt, tolower, word_tokenizer, chunks_number = 10, ids = ids) # Example of stemming tokenizer # stem_tokenizer = function(x) { # word_tokenizer(x) %>% lapply(SnowballC::wordStem('en')) # }