class
BERTTokenizer
extends
WordPieceTokenizerBERTTokenizer(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)BERT tokenizer — pure-Python reference.
A thin convenience subclass of
lucid.utils.tokenizer.WordPieceTokenizer with BERT's
canonical [UNK]/[CLS]/[SEP]/[PAD]/[MASK] registry, the
lucid.utils.tokenizer.normalizers.BERTNormalizer
(lowercase + accent-strip + Chinese-char split + clean-text), and
lucid.utils.tokenizer.pre_tokenizers.WhitespacePunctuationSplit
pre-tokenizer baked in as defaults.
Loads any published HF BERT-family vocab.txt checkpoint
unchanged.
Parameters
vocabdict[str, int]BERT-style vocab. Continuation subwords prefixed with
##.
Pass {} and call WordPieceTokenizer.train to
train a fresh vocab from a corpus.unk_tokenstr= "[UNK]"Token emitted when no valid longest-match prefix is found.
Must be present in
vocab for the id to materialise.continuing_prefixstr= "##"Marker for non-initial subwords (BERT convention).
max_chars_per_wordint= 100Words longer than this skip the longest-match loop and emit
the
unk_token directly (mirrors BERT BasicTokenizer).do_lower_casebool= TrueForwarded to
BERTNormalizer. Set to False for
cased checkpoints (bert-base-cased, multilingual models).
Ignored if normalizer is given explicitly.normalizerNormalizer= NoneOverride the default
BERTNormalizer.pre_tokenizerPreTokenizer= NoneOverride the default
WhitespacePunctuationSplit.Override the default BERT registry.
Examples
Train a small BERT-style tokenizer from a corpus, then round-trip:
>>> from lucid.models.text.bert import BERTTokenizer
>>> tok = BERTTokenizer(vocab={})
>>> tok.train(["hello world", "hello bert"] * 8, vocab_size=60)
>>> ids = tok.encode("Hello World", add_special_tokens=False)
>>> # default do_lower_case=True → "Hello" and "hello" share ids.
>>> tok.decode(ids, skip_special_tokens=False)
'hello world'See Also
BERTTokenizerFast—C++-backed variant with identical output.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, do_lower_case: bool = True, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)