class
BERTTokenizerFast
extends
WordPieceTokenizerFastBERTTokenizerFast(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)BERT tokenizer — C++-backed.
Bit-identical to BERTTokenizer: same vocab format, same
special-token registry, same encode output. The greedy
longest-match loop runs in C++ via the engine-side WordPiece
binding, typically 20-30x faster than the pure-Python flavour on
large corpora.
Constructor parameters mirror BERTTokenizer exactly —
see that class for the full reference.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)