class
BERTTokenizerFast
extends
WordPieceTokenizerFastBERTTokenizerFast(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)BERT tokenizer — C++-backed.
Bit-identical to BERTTokenizer: same vocab format, same
special-token registry, same encode output. The greedy
longest-match loop runs in C++ via the engine-side WordPiece
binding, typically 20-30x faster than the pure-Python flavour on
large corpora.
Constructor parameters mirror BERTTokenizer exactly —
see that class for the full reference.
Examples
>>> from lucid.models.text.bert._tokenizer import BERTTokenizerFast
>>> vocab = {"[UNK]": 0, "he": 1, "##llo": 2}
>>> BERTTokenizerFast(vocab=vocab).encode("hello")
[1, 2]
The same ids the plain tokenizer produces. The speed is in the
lookup, and a pair that segmented differently would be a bug no type
can catch.Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)