Abstract base class for all key/value caches used in attention.
A cache stores, per transformer layer, the key and value projections of every token seen so far so that an incremental (one-token-at-a-time) forward pass only has to project the new token's key/value and attend against the accumulated history — turning an re-encode of the whole prefix at every step into an update.
Subclasses implement update (the per-layer write/read), plus the
length-introspection and beam-reordering hooks. Concrete subclasses are
lucid.utils.cache.DynamicCache (the default growing cache),
lucid.utils.cache.StaticCache (a fixed pre-allocated buffer for
compiled decoding), and lucid.utils.cache.EncoderDecoderCache
(paired self- + cross-attention).
Used by 4
Instance methods
5Return the maximum sequence length the cache can hold.
None for an unbounded (dynamically growing) cache.
Return the number of cached tokens for layer_idx (0 if empty).
Return how many cached tokens can be used given new_seq_length.
For a bounded cache this clamps so that previous + new never
exceeds get_max_cache_shape; for an unbounded cache it is
simply the current sequence length.
Parameters
new_seq_lengthintlayer_idxint= 0Returns
intThe count of already-cached tokens that remain usable once the
new_seq_length tokens are accounted for against the cap.
Reindex the cache along the batch dimension with beam_idx.
Used by beam search to keep each beam's cache aligned after the per-step beam permutation. Mutates the cache in place.
update(key_states: Tensor, value_states: Tensor, layer_idx: int, cache_kwargs: dict[str, object] | None = None)Append key_states / value_states for layer_idx and return
the full cached key/value tensors.
Parameters
key_statesTensor(B, num_heads, T_new, head_dim).value_statesTensorlayer_idxintcache_kwargsdict or None= Nonecache_position).
Ignored by the dynamic cache; reserved for parity with other
cache implementations.