cache
4 memberslucid.utils.cachelucid.utils.cache: key/value caches for incremental attention (generation).
Class names, method names, and argument names mirror the reference
cache_utils module so that generation code ports across with only an
import change. The eager default is DynamicCache (grows by
concatenation); StaticCache is a fixed pre-allocated buffer that keeps
shapes constant for compiled single-token decoding; and encoder-decoder models
use EncoderDecoderCache (paired self- + cross-attention). Canonical
paths: lucid.utils.cache.Cache / DynamicCache / StaticCache /
EncoderDecoderCache.
Classes
Cache5 methodsAbstract base class for all key/value caches used in attention.
DynamicCache15 methodsA cache that grows its key/value tensors by concatenation each step.
EncoderDecoderCache14 methodsWraps a decoder self-attention cache and a cross-attention cache.
StaticCache13 methodsA bounded cache backed by fixed pre-allocated buffers written in place.