wordpiece (version 2.1.3)

.process_wp_vocab: Process a Wordpiece Vocabulary for Tokenization

Description

Process a Wordpiece Vocabulary for Tokenization

Usage

.process_wp_vocab(v)
# S3 method for default
.process_wp_vocab(v)
# S3 method for wordpiece_vocabulary
.process_wp_vocab(v)
# S3 method for integer
.process_wp_vocab(v)
# S3 method for character
.process_wp_vocab(v)

Arguments

v

An object of class wordpiece_vocabulary.

Value

A character vector of tokens for tokenization.