TextEmbeddingModel: Text embedding model

Description

This R6 class stores a text embedding model which can be used to tokenize, encode, decode, and embed raw texts. The object provides a unique interface for different text processing methods.

Arguments

Value

Objects of class TextEmbeddingModel transform raw texts into numerical representations which can be used for downstream tasks. For this aim objects of this class allow to tokenize raw texts, to encode tokens to sequences of integers, and to decode sequences of integers back to tokens.

Public fields

last_training: ('list()')
List for storing the history and the results of the last training. This information will be overwritten if a new training is started.

Methods

Public methods

Method `new()`

Method for creating a new text embedding model

Usage

TextEmbeddingModel$new(
  model_name = NULL,
  model_label = NULL,
  model_version = NULL,
  model_language = NULL,
  method = NULL,
  ml_framework = aifeducation_config$get_framework()$TextEmbeddingFramework,
  max_length = 0,
  chunks = 1,
  overlap = 0,
  emb_layer_min = "middle",
  emb_layer_max = "2_3_layer",
  emb_pool_type = "average",
  model_dir,
  bow_basic_text_rep,
  bow_n_dim = 10,
  bow_n_cluster = 100,
  bow_max_iter = 500,
  bow_max_iter_cluster = 500,
  bow_cr_criterion = 1e-08,
  bow_learning_rate = 1e-08,
  trace = FALSE
)

Arguments

model_name: string containing the name of the new model.

model_label

string containing the label/title of the new model.

model_version

string version of the model.

model_language

string containing the language which the model represents (e.g., English).

method

string determining the kind of embedding model. Currently the following models are supported: method="bert" for Bidirectional Encoder Representations from Transformers (BERT), method="roberta" for A Robustly Optimized BERT Pretraining Approach (RoBERTa), method="longformer" for Long-Document Transformer, method="funnel" for Funnel-Transformer, method="deberta_v2" for Decoding-enhanced BERT with Disentangled Attention (DeBERTa V2), method="glove" for GlobalVector Clusters, and method="lda" for topic modeling. See details for more information.

ml_framework

string Framework to use for the model. ml_framework="tensorflow" for 'tensorflow' and ml_framework="pytorch" for 'pytorch'. Only relevant for transformer models.

max_length

int determining the maximum length of token sequences used in transformer models. Not relevant for the other methods.

chunks

int Maximum number of chunks. Only relevant for transformer models.

overlap

int determining the number of tokens which should be added at the beginning of the next chunk. Only relevant for BERT models.

emb_layer_min

int or string determining the first layer to be included in the creation of embeddings. An integer correspondents to the layer number. The first layer has the number 1. Instead of an integer the following strings are possible: "start" for the first layer, "middle" for the middle layer, "2_3_layer" for the layer two-third layer, and "last" for the last layer.

emb_layer_max

int or string determining the last layer to be included in the creation of embeddings. An integer correspondents to the layer number. The first layer has the number 1. Instead of an integer the following strings are possible: "start" for the first layer, "middle" for the middle layer, "2_3_layer" for the layer two-third layer, and "last" for the last layer.

emb_pool_type

string determining the method for pooling the token embeddings within each layer. If "cls" only the embedding of the CLS token is used. If "average" the token embedding of all tokens are averaged (excluding padding tokens).

model_dir

string path to the directory where the BERT model is stored.

bow_basic_text_rep

object of class basic_text_rep created via the function bow_pp_create_basic_text_rep. Only relevant for method="glove_cluster" and method="lda".

bow_n_dim

int Number of dimensions of the GlobalVector or number of topics for LDA.

bow_n_cluster

int Number of clusters created on the basis of GlobalVectors. Parameter is not relevant for method="lda" and method="bert"

bow_max_iter

int Maximum number of iterations for fitting GlobalVectors and Topic Models.

bow_max_iter_cluster

int Maximum number of iterations for fitting cluster if method="glove".

bow_cr_criterion

double convergence criterion for GlobalVectors.

bow_learning_rate

double initial learning rate for GlobalVectors.

trace

bool TRUE prints information about the progress. FALSE does not.

Details

method: In the case of method="bert", method="roberta", and method="longformer", a pretrained transformer model must be supplied via model_dir. For method="glove" and method="lda" a new model will be created based on the data provided via bow_basic_text_rep. The original algorithm for GlobalVectors provides only word embeddings, not text embeddings. To achieve text embeddings the words are clustered based on their word embeddings with kmeans.

Returns

Returns an object of class TextEmbeddingModel.

Method `load_model()`

Method for loading a transformers model into R.

Usage

TextEmbeddingModel$load_model(model_dir, ml_framework = "auto")

Arguments

model_dir: string containing the path to the relevant model directory.

ml_framework

string Determines the machine learning framework for using the model. Possible are ml_framework="pytorch" for 'pytorch', ml_framework="tensorflow" for 'tensorflow', and ml_framework="auto".

Returns

Function does not return a value. It is used for loading a saved transformer model into the R interface.

Method `save_model()`

Method for saving a transformer model on disk.Relevant only for transformer models.

Usage

TextEmbeddingModel$save_model(model_dir, save_format = "default")

Arguments

model_dir: string containing the path to the relevant model directory.

save_format

Format for saving the model. For 'tensorflow'/'keras' models "h5" for HDF5. For 'pytorch' models "safetensors" for 'safetensors' or "pt" for 'pytorch' via pickle. Use "default" for the standard format. This is h5 for 'tensorflow'/'keras' models and safetensors for 'pytorch' models.

Returns

Function does not return a value. It is used for saving a transformer model to disk.

Method `encode()`

Method for encoding words of raw texts into integers.

Usage

TextEmbeddingModel$encode(
  raw_text,
  token_encodings_only = FALSE,
  to_int = TRUE,
  trace = FALSE
)

Arguments

raw_text: vector containing the raw texts.

token_encodings_only

bool If TRUE, only the token encodings are returned. If FALSE, the complete encoding is returned which is important for BERT models.

to_int

bool If TRUE the integer ids of the tokens are returned. If FALSE the tokens are returned. Argument only applies for transformer models and if token_encodings_only==TRUE.

trace

bool If TRUE, information of the progress is printed. FALSE if not requested.

Returns

list containing the integer sequences of the raw texts with special tokens.

Method `decode()`

Method for decoding a sequence of integers into tokens

Usage

TextEmbeddingModel$decode(int_seqence, to_token = FALSE)

Arguments

int_seqence: list containing the integer sequences which should be transformed to tokens or plain text.

to_token

bool If FALSE a plain text is returned. if TRUE a sequence of tokens is returned. Argument only relevant if the model is based on a transformer.

Returns

list of token sequences

Method `get_special_tokens()`

Method for receiving the special tokens of the model

Usage

TextEmbeddingModel$get_special_tokens()

Returns

Returns a matrix containing the special tokens in the rows and their type, token, and id in the columns.

Method `embed()`

Method for creating text embeddings from raw texts

In the case of using a GPU and running out of memory reduce the batch size or restart R and switch to use cpu only via set_config_cpu_only.

Usage

TextEmbeddingModel$embed(
  raw_text = NULL,
  doc_id = NULL,
  batch_size = 8,
  trace = FALSE
)

Arguments

raw_text: vector containing the raw texts.

doc_id

vector containing the corresponding IDs for every text.

batch_size

int determining the maximal size of every batch.

trace

bool TRUE, if information about the progression should be printed on console.

Returns

Method returns a R6 object of class EmbeddedText. This object contains the embeddings as a data.frame and information about the model creating the embeddings.

Method `fill_mask()`

Method for calculating tokens behind mask tokens.

Usage

TextEmbeddingModel$fill_mask(text, n_solutions = 5)

Arguments

text: string Text containing mask tokens.

n_solutions

int Number estimated tokens for every mask.

Returns

Returns a list containing a data.frame for every mask. The data.frame contains the solutions in the rows and reports the score, token id, and token string in the columns.

Method `set_publication_info()`

Method for setting the bibliographic information of the model.

Usage

TextEmbeddingModel$set_publication_info(type, authors, citation, url = NULL)

Arguments

type: string Type of information which should be changed/added. type="developer", and type="modifier" are possible.

authors

List of people.

citation

string Citation in free text.

url

string Corresponding URL if applicable.

Returns

Function does not return a value. It is used to set the private members for publication information of the model.

Method `get_publication_info()`

Method for getting the bibliographic information of the model.

Usage

TextEmbeddingModel$get_publication_info()

Returns

list of bibliographic information.

Method `set_software_license()`

Method for setting the license of the model

Usage

TextEmbeddingModel$set_software_license(license = "GPL-3")

Arguments

license: string containing the abbreviation of the license or the license text.

Returns

Function does not return a value. It is used for setting the private member for the software license of the model.

Method `get_software_license()`

Method for requesting the license of the model

Usage

TextEmbeddingModel$get_software_license()

Returns

string License of the model

Method `set_documentation_license()`

Method for setting the license of models' documentation.

Usage

TextEmbeddingModel$set_documentation_license(license = "CC BY-SA")

Arguments

license: string containing the abbreviation of the license or the license text.

Returns

Function does not return a value. It is used to set the private member for the documentation license of the model.

Method `get_documentation_license()`

Method for getting the license of the models' documentation.

Usage

TextEmbeddingModel$get_documentation_license()

Arguments

license: string containing the abbreviation of the license or the license text.

Method `set_model_description()`

Method for setting a description of the model

Usage

TextEmbeddingModel$set_model_description(
  eng = NULL,
  native = NULL,
  abstract_eng = NULL,
  abstract_native = NULL,
  keywords_eng = NULL,
  keywords_native = NULL
)

Arguments

eng: string A text describing the training of the classifier, its theoretical and empirical background, and the different output labels in English.

native

string A text describing the training of the classifier, its theoretical and empirical background, and the different output labels in the native language of the model.

abstract_eng

string A text providing a summary of the description in English.

abstract_native

string A text providing a summary of the description in the native language of the classifier.

keywords_eng

vector of keywords in English.

keywords_native

TextEmbeddingModel$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.

Description

Arguments

Value

Public fields

Methods

Public methods

Method new()

Usage

Arguments

Details

Returns

Method load_model()

Usage

Arguments

Returns

Method save_model()

Usage

Arguments

Returns

Method encode()

Usage

Arguments

Returns

Method decode()

Usage

Arguments

Returns

Method get_special_tokens()

Usage

Returns

Method embed()

Usage

Arguments

Returns

Method fill_mask()

Usage

Arguments

Returns

Method set_publication_info()

Usage

Arguments

Returns

Method get_publication_info()

Usage

Returns

Method set_software_license()

Usage

Arguments

Returns

Method get_software_license()

Usage

Returns

Method set_documentation_license()

Usage

Arguments

Returns

Method get_documentation_license()

Usage

Arguments

Method set_model_description()

Usage

Arguments

Returns

Method get_model_description()

Usage

Returns

Method get_model_info()

Usage

Returns

Method get_package_versions()

Usage

Returns

Method get_basic_components()

Usage

Returns

Method get_bow_components()

Usage

Returns

Method get_transformer_components()

Usage

Method `new()`

Method `load_model()`

Method `save_model()`

Method `encode()`

Method `decode()`

Method `get_special_tokens()`

Method `embed()`

Method `fill_mask()`

Method `set_publication_info()`

Method `get_publication_info()`

Method `set_software_license()`

Method `get_software_license()`

Method `set_documentation_license()`

Method `get_documentation_license()`

Method `set_model_description()`

Method `get_model_description()`

Method `get_model_info()`

Method `get_package_versions()`

Method `get_basic_components()`

Method `get_bow_components()`

Method `get_transformer_components()`

Method `get_sustainability_data()`

Method `get_ml_framework()`

Method `clone()`