Abstract base for every tokenizer in lucid.utils.tokenizer.
Subclasses implement four hooks:
_encode_one(text) -> list[int]— algorithm-specific encode for one pre-normalized + pre-tokenized chunk. The base class handles normalization, pre-tokenization, and special-token insertion around this hook._decode_one(ids) -> str— algorithm-specific decode of one id sequence (special tokens NOT yet stripped — the base class handlesskip_special_tokens).vocab_size— total number of distinct ids.algo— short algorithm name for serialisation ("bpe","wordpiece","unigram","byte_bpe").
The base class layers
encode / decode / encode_batch /
decode_batch / __call__ on top — those are
fully generic and don't need to be overridden.
See Also
algorithm with identical encode outputs.
Used by 9
Constructors
2Initialise the special-token registry; subclasses must populate the vocab + algorithm state before calling this.
__call__
→dict__call__(text: str | list[str], add_special_tokens: bool = True, padding: bool | str = False, truncation: bool = False, max_length: int | None = None, return_tensors: str | None = None, return_attention_mask: bool = True)Hugging-Face-style entry point.
Parameters
textstr or list of stradd_special_tokensbool= Truepaddingbool or str= FalseFalse — no padding. True or "longest" — pad
every sequence to the longest in the batch.
"max_length" — pad every sequence to max_length.truncationbool= FalseTrue, truncate each sequence past
max_length (no-op if max_length is None).max_lengthint= Nonepadding="max_length".return_tensorslucid= "lucid""lucid", wrap each output value as a
lucid.Tensor (dtype int32) and stack the batch
into a 2-D tensor. When None, return plain Python
lists.return_attention_maskbool= TrueTrue (and padding is set), include an
"attention_mask" entry: 1 for real tokens, 0 for
padding.Returns
dictAlways contains "input_ids". Optionally
"attention_mask" (when padding + return_attention_mask)
and "special_tokens_mask" (always, for downstream
BERT-style MLM consumers).
Properties
11Short algorithm name ("bpe" / "wordpiece" / ...).
Used by save to route serialisation to the right
format and by from_pretrained to pick the right
subclass when loading from a generic path.
Every special-token id (canonical + extras), deduped.
Id of the bos special token (None if undefined).
Id of the cls special token (None if undefined).
Id of the eos special token (None if undefined).
Id of the mask special token (None if undefined).
Id of the pad special token (None if undefined).
Id of the sep special token (None if undefined).
special_tokens: SpecialTokensThe registry of canonical + extra special tokens.
Id of the unk special token (None if undefined).
Total number of distinct token ids (including specials).
Instance methods
9Inverse of convert_tokens_to_ids.
Map one or many token strings to their ids.
Unknown tokens raise KeyError (callers wanting an
UNK fallback should do their own .get(token, unk_id)
lookup).
Decode a sequence of ids back to text.
When skip_special_tokens=True every id in
all_special_ids is dropped before the
algorithm-specific _decode_one runs.
Parameters
idsiterable of intskip_special_tokens(bool, optional, keyword - only)= TrueTrue, every id in all_special_ids
(BOS/EOS/PAD/CLS/SEP/MASK + any extras) is filtered out
before _decode_one reassembles the surface string.Returns
strReconstructed text — exact reverse of encode for
lossless algorithms (Unigram / byte-BPE); lossy for
normalising flavours (NFC + Lowercase).
decode_batch
→list[str]decode_batch(batch_ids: Iterable[Iterable[int]] | list[list[int]], skip_special_tokens: bool = True)Vectorised decode over a list of id sequences.
Encode a single string to a list of token ids.
When add_special_tokens=True the canonical wrappers
(BOS/EOS for autoregressive, CLS/SEP for BERT-style) are
injected around the algorithm output — subclasses with
non-standard wrapping (e.g. T5's </s> only at the end)
override _build_inputs_with_special_tokens.
Parameters
textstr_encode_one hook before the BPE / WordPiece /
Unigram merge loop runs.add_special_tokens(bool, optional, keyword - only)= TrueTrue, prepend / append the algorithm's special
tokens (BOS/EOS for autoregressive families, CLS/SEP for
BERT-style) via
_build_inputs_with_special_tokens. Pass False
to return the raw merge-loop output untouched (useful for
offline analysis, alignment, or batching pipelines that
inject specials elsewhere).Returns
list of intToken ids for text in left-to-right order.
encode_batch
→list of list of intencode_batch(texts: Iterable[str] | list[str], add_special_tokens: bool = True)Vectorised encode over a list of strings.
Default loops through encode; subclasses with a true
batched fast path (the C++ Fast tokenizers) override.
Parameters
textsiterable of stradd_special_tokens(bool, optional, keyword - only)= Trueencode for each item — when True,
every sequence is wrapped with its canonical BOS/EOS or
CLS/SEP markers.Returns
list of list of intOne id sequence per input string, same order as texts.
Return the token-string → id map.
Default returns an empty dict; subclasses with a real vocab
override this. Used by save and by special-token-id
resolution in _refresh_special_ids.
Return the surface form for token_id or None if
out of range.
Default does an O(N) reverse scan of get_vocab;
subclasses with an O(1) reverse table should override.
Persist the tokenizer to directory.
Default writes tokenizer.json (unified format) +
special_tokens_map.json. Subclasses with algorithm-
specific legacy files (vocab.json + merges.txt for
BPE, vocab.txt for WordPiece) override this hook to
emit those alongside.