class
RoFormerTokenizer
extends
WordPieceTokenizerRoFormerTokenizer(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)RoFormer tokenizer — pure-Python reference.
Identical algorithm to lucid.models.text.bert.BERTTokenizer
(WordPiece longest-match with [UNK]/[CLS]/[SEP]/[PAD]/[MASK]
registered out of the box). Kept as a separate class so every
text-model family exposes a uniform
from lucid.models.text.<family> import {Family}Tokenizer
surface — useful for generic model-loading helpers.
Parameters
vocabdict[str, int]BERT-style vocab with
## continuation prefix.unk_tokenstr= "[UNK]"No description.
continuing_prefixstr= "##"No description.
max_chars_per_wordint= 100No description.
do_lower_casebool= TrueSet
False for cased RoFormer checkpoints.normalizerNormalizer= NoneNo description.
pre_tokenizerPreTokenizer= NoneNo description.
No description.
See Also
RoFormerTokenizerFast—C++-backed variant with identical output.- lucid.models.text.bert.BERTTokenizer—Sibling wrapper sharing
the same algorithm and defaults.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)