textrecipes (version 1.0.6)

tokenlist: Create Token Object

Description

A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.

Usage

tokenlist(tokens = list(), lemma = NULL, pos = NULL)

Value

a tokenlist object.

Arguments

tokens

List of character vectors

lemma

List of character vectors, must be same size and shape as x.

pos

List of character vectors, must be same size and shape as x.

Examples

abc <- list(letters, LETTERS)
tokenlist(abc)

unclass(tokenlist(abc))

tibble(text = tokenlist(abc))

library(tokenizers)
library(modeldata)
data(tate_text)
tokens <- tokenize_words(as.character(tate_text$medium))

tokenlist(tokens)