Abstract class for all tokenizers used with the 'transformers' library.
Does return a new object of this class.
aifeducation::AIFEMaster -> aifeducation::TokenizerBase -> HuggingFaceTokenizer
Inherited methods
aifeducation::AIFEMaster$get_all_fields()aifeducation::AIFEMaster$get_documentation_license()aifeducation::AIFEMaster$get_ml_framework()aifeducation::AIFEMaster$get_model_config()aifeducation::AIFEMaster$get_model_description()aifeducation::AIFEMaster$get_model_info()aifeducation::AIFEMaster$get_model_license()aifeducation::AIFEMaster$get_package_versions()aifeducation::AIFEMaster$get_private()aifeducation::AIFEMaster$get_publication_info()aifeducation::AIFEMaster$get_sustainability_data()aifeducation::AIFEMaster$is_configured()aifeducation::AIFEMaster$is_trained()aifeducation::AIFEMaster$set_documentation_license()aifeducation::AIFEMaster$set_model_description()aifeducation::AIFEMaster$set_model_license()aifeducation::AIFEMaster$set_publication_info()aifeducation::TokenizerBase$calculate_statistics()aifeducation::TokenizerBase$decode()aifeducation::TokenizerBase$encode()aifeducation::TokenizerBase$get_special_tokens()aifeducation::TokenizerBase$get_tokenizer()aifeducation::TokenizerBase$get_tokenizer_statistics()aifeducation::TokenizerBase$load_from_disk()aifeducation::TokenizerBase$n_special_tokens()aifeducation::TokenizerBase$save()
create_from_hf()Creates a tokenizer from a pretrained model
HuggingFaceTokenizer$create_from_hf(model_dir)model_dir
Does return a new object of this class.
clone()The objects of this class are cloneable with this method.
HuggingFaceTokenizer$clone(deep = FALSE)deepWhether to make a deep clone.
Other Tokenizer:
WordPieceTokenizer