quanteda (version 4.5.0)

tokens_recompile: Remove redundant or unused tokens IDs

Description

This function reassign tokens IDs after changes in removal or transformation of tokens. It update token IDs to ensure that they are unique and dense. Usually, this function is only applied to tokens_xptr objects at the end of text pre-processing.

Usage

tokens_recompile(x, force = FALSE)

Arguments

x

the tokens_xptr object.

force

if TRUE, update tokens IDs even when it may not be necessary.

Examples

toks <- tokens(c(one = "a b c d A B C D",
                 two = "A B C d"), xptr = TRUE)
toks <- tokens_tolower(toks)
types(toks)
types(tokens_recompile(toks))