model_wordpiece

<p>An implementation of the WordPiece algorithm</p>
<p>An implementation of the WordPiece algorithm</p>

Interfaces with the 'Hugging Face' tokenizers library to provide implementations
of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm
<https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both
training new vocabularies and tokenizing texts.

Daniel Falbel

Fast Text Tokenization

model_wordpiece function

<p><code>tok::tok_model</code> -&gt; <code>tok_model_wordpiece</code></p>

Super class


<div class="section" id="public-methods">
<h3>Public methods</h3><p></p><p><ul>
<li><p><a href="/link/model_wordpiece%24new()?package=tok&version=0.2.1" data-mini-rdoc="tok::model_wordpiece$new()"><code>model_wordpiece$new()</code></a></p></li>
<li><p><a href="/link/model_wordpiece%24clone()?package=tok&version=0.2.1" data-mini-rdoc="tok::model_wordpiece$clone()"><code>model_wordpiece$clone()</code></a></p></li>
</ul></p><p></p></div><p><hr>
<a id="method-tok_model_wordpiece-new"></a></p><div class="section" id="method-new-">
<h3>Method <code>new()</code></h3>
<p>Constructor for the wordpiece tokenizer</p><div class="section" id="usage">
<h4>Usage</h4>
<p><div class="r"></div></p><pre><code>model_wordpiece$new(
  vocab = NULL,
  unk_token = NULL,
  max_input_chars_per_word = NULL
)</code></pre><p></p></div><p></p>
</div><p></p><p><div class="section" id="arguments">
<h4>Arguments</h4>
<p><div class="arguments"></div></p><dl>
<dt><code>vocab</code></dt>
<dd><p>A dictionary of string keys and their corresponding ids.
Default: <code>NULL</code>.</p></dd></dl></div></p><p>
<dt><code>unk_token</code></dt>
<dd><p>The unknown token to be used by the model.
Default: <code>NULL</code>.</p></dd></p><p>
<dt><code>max_input_chars_per_word</code></dt>
<dd><p>The maximum number of characters to allow in a single word.
Default: <code>NULL</code>.</p></dd></p><p>
</p><p></p>
<p></p><p></p><p><hr>
<a id="method-tok_model_wordpiece-clone"></a></p><div class="section" id="method-clone-">
<h3>Method <code>clone()</code></h3>
<p>The objects of this class are cloneable with this method.</p><div class="section" id="usage">
<h4>Usage</h4>
<p><div class="r"></div></p><pre><code>model_wordpiece$clone(deep = FALSE)</code></pre><p></p></div><p></p>
</div><p></p><p><div class="section" id="arguments">
<h4>Arguments</h4>
<p><div class="arguments"></div></p><dl>
<dt><code>deep</code></dt>
<dd><p>Whether to make a deep clone.</p></dd></dl></div></p><p>
</p><p></p>
<p></p><p>
</p>

Methods

An implementation of the WordPiece algorithm — model_wordpiece


<div class='section' id='public-methods'>
<h3>Public methods</h3>

<ul>
<li><p><a href='#method-tok_model_wordpiece-new'><code>model_wordpiece$new()</code></a></p></li>
<li><p><a href='#method-tok_model_wordpiece-clone'><code>model_wordpiece$clone()</code></a></p></li>
</ul>

</div><p><hr>
<a id="method-tok_model_wordpiece-new"></a></p><div class='section' id='method-new-'>
<h3>Method <code>new()</code></h3>
<p>Constructor for the wordpiece tokenizer</p><div class='section' id='usage'>
<h4>Usage</h4>
<p><div class="r"></p><pre><code>model_wordpiece$new(
  vocab = NULL,
  unk_token = NULL,
  max_input_chars_per_word = NULL
)</code></pre><p></div></p>
</div>

<div class='section' id='arguments'>
<h4>Arguments</h4>
<p><div class="arguments"></p><dl>
<dt><code>vocab</code></dt>
<dd><p>A dictionary of string keys and their corresponding ids.
Default: <code>NULL</code>.</p></dd>


<dt><code>unk_token</code></dt>
<dd><p>The unknown token to be used by the model.
Default: <code>NULL</code>.</p></dd>


<dt><code>max_input_chars_per_word</code></dt>
<dd><p>The maximum number of characters to allow in a single word.
Default: <code>NULL</code>.</p></dd>


</dl><p></div></p>
</div>

</div><p><hr>
<a id="method-tok_model_wordpiece-clone"></a></p><div class='section' id='method-clone-'>
<h3>Method <code>clone()</code></h3>
<p>The objects of this class are cloneable with this method.</p><div class='section' id='usage'>
<h4>Usage</h4>
<p><div class="r"></p><pre><code>model_wordpiece$clone(deep = FALSE)</code></pre><p></div></p>
</div>

<div class='section' id='arguments'>
<h4>Arguments</h4>
<p><div class="arguments"></p><dl>
<dt><code>deep</code></dt>
<dd><p>Whether to make a deep clone.</p></dd>


</dl><p></div></p>
</div>

</div>


model_wordpiece: An implementation of the WordPiece algorithm

Description

Arguments

Super class

Methods

Public methods

Method `new()`

Usage

Arguments

Method `clone()`

Usage

Arguments

See Also

Description

Arguments

Super class

Methods

Public methods

Method new()

Usage

Arguments

Method clone()

Usage

Arguments

See Also

Method `new()`

Method `clone()`