.mp_tokenize_single_string

<p>Character; a vector of words (generated by space-tokenizing a
single input).</p>

words

vocab

lookup

unk_token

<p>Maximum length of word recognized.</p>

max_chars

internal

Tokenize text into morphemes. The morphemepiece algorithm uses a
lookup table to determine the morpheme breakdown of words, and falls back on a
modified wordpiece tokenization algorithm for words not found in the lookup
table.

Jonathan Bratt

morphemepiece

Morpheme Tokenization

Jon Harmon

Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning 

.mp_tokenize_single_string function

Tokenize an Input Word-by-word — .mp_tokenize_single_string

Tokenize an Input Word-by-word

.mp_tokenize_single_string: Tokenize an Input Word-by-word

Description

Usage

Arguments

Value