fastrtext (version 0.3.3)

get_tokenized_text: Tokenize text

Description

Separate words in a text using space characters

Usage

get_tokenized_text(model, texts)

Arguments

model

fastText model

texts

a character containing the documents

Value

a list of character containing words

Examples

# NOT RUN {
library(fastrtext)
model_test_path <- system.file("extdata", "model_unsupervised_test.bin", package = "fastrtext")
model <- load_model(model_test_path)
tokens <- get_tokenized_text(model, "this is a test")
print(tokens)
tokens <- get_tokenized_text(model, c("this is a test 1", "this is a second test!"))
print(tokens)
# }