textrecipes (version 0.5.1)

tokenlist: Create Token Object

Description

A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.

Usage

tokenlist(tokens = list(), lemma = NULL, pos = NULL)

Arguments

tokens

List of character vectors

lemma

List of character vectors, must be same size and shape as x.

pos

List of character vectors, must be same size and shape as x.

Value

a tokenlist object.

Examples

# NOT RUN {
abc <- list(letters, LETTERS)
tokenlist(abc)

unclass(tokenlist(abc))

tibble(text = tokenlist(abc))

library(tokenizers)
library(modeldata)
data(tate_text)
tokens <- tokenize_words(as.character(tate_text$medium))

tokenlist(tokens)
# }