class
RoFormerTokenizerFast
extends
WordPieceTokenizerFastRoFormerTokenizerFast(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)RoFormer tokenizer — C++-backed.
Bit-identical to RoFormerTokenizer. The greedy
longest-match loop runs in C++ via the engine-side WordPiece
binding.
Constructor parameters mirror RoFormerTokenizer — see
that class for the full reference.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)