create_dtm
From text2vec v0.3.0
by Dmitriy Selivanov
Document-term matrix construction
This is a high-level function for creating a document-term matrix. If a parallel backend is registered, it will construct the DTM in multiple threads.
Usage
create_dtm(itoken_src, vectorizer, type = c("dgCMatrix", "dgTMatrix", "lda_c"), ...)
"create_dtm"(itoken_src, vectorizer, type = c("dgCMatrix", "dgTMatrix", "lda_c"), ...)
"create_dtm"(itoken_src, vectorizer, type = c("dgCMatrix", "dgTMatrix", "lda_c"), verbose = FALSE, ...)
Arguments
- itoken_src
list
of iterators over tokens provided byitoken
. Each element is a list of tokens, that is, tokenized and normalized strings.- vectorizer
function
vectorizer function; see vectorizers.- type
character
, one ofc("dgCMatrix", "dgTMatrix", "lda_c")
."lda_c"
is Blei's lda-c format (a list of 2 * doc_terms_size); see https://www.cs.princeton.edu/~blei/lda-c/readme.txt- ...
- arguments to the foreach function which is used to iterate
over
itoken_src
. - verbose
logical
print status messages
Details
The user should keep in mind that he or she should split the data
itself and provide a list of itoken iterators. Each element of
itoken_src
will be handled in separate thread and combined at the
end of processing.
Value
-
A document-term matrix
See Also
Examples
## Not run:
# data("movie_review")
# N <- 1000
# it <- itoken(movie_review$review[1:N], preprocess_function = tolower,
# tokenizer = word_tokenizer)
# v <- create_vocabulary(it)
# #remove very common and uncommon words
# pruned_vocab = prune_vocabulary(v, term_count_min = 10,
# doc_proportion_max = 0.5, doc_proportion_min = 0.001)
# vectorizer <- vocab_vectorizer(v)
# it <- itoken(movie_review$review[1:N], preprocess_function = tolower,
# tokenizer = word_tokenizer)
# dtm <- create_dtm(it, vectorizer)
# # get tf-idf matrix from bag-of-words matrix
# dtm_tfidf <- transformer_tfidf(dtm)
#
# ## Example of parallel mode
# # set to number of cores on your machine
# N_WORKERS <- 1
# doParallel::registerDoParallel(N_WORKERS)
# splits <- split_into(movie_review$review, N_WORKERS)
# jobs <- lapply(splits, itoken, tolower, word_tokenizer, chunks_number = 1)
# vectorizer <- hash_vectorizer()
# dtm <- create_dtm(jobs, vectorizer, type = 'dgTMatrix')
# ## End(Not run)
Community examples
Looks like there are no examples yet.