Lucid
/

Tensor

  • lucid.Tensor
  • lucid.tensor
  • Tensor Operations
    • lucid.exp
    • lucid.log
    • lucid.log2
    • lucid.sqrt
    • lucid.sin
    • lucid.cos
    • lucid.tan
    • lucid.arcsin
    • lucid.arccos
    • lucid.arctan
    • lucid.sinh
    • lucid.cosh
    • lucid.tanh
    • lucid.clip
    • lucid.abs
    • lucid.sign
    • lucid.reciprocal
    • lucid.square
    • lucid.cube
    • lucid.transpose
    • lucid.sum
    • lucid.trace
    • lucid.mean
    • lucid.var
    • lucid.min
    • lucid.max
    • lucid.swapaxes
    • lucid.round
    • lucid.floor
    • lucid.ceil
    • lucid.cumprod
    • lucid.cumsum
    • lucid.add
    • lucid.sub
    • lucid.multiply
    • lucid.div
    • lucid.minimum
    • lucid.maximum
    • lucid.power
    • lucid.dot
    • lucid.inner
    • lucid.outer
    • lucid.matmul
    • lucid.tensordot
    • lucid.zeros
    • lucid.zeros_like
    • lucid.ones
    • lucid.ones_like
    • lucid.eye
    • lucid.diag
    • lucid.arange
    • lucid.empty
    • lucid.empty_like
    • lucid.linspace
    • lucid.full
    • lucid.full_like
  • Tensor Utilities
    • lucid.reshape
    • lucid.squeeze
    • lucid.unsqueeze
    • lucid.ravel
    • lucid.stack
    • lucid.hstack
    • lucid.vstack
    • lucid.concatenate
    • lucid.pad
    • lucid.repeat
    • lucid.tile
    • lucid.flatten
    • lucid.meshgrid
    • lucid.split
    • lucid.tril
    • lucid.triu
    • lucid.broadcast_to
    • lucid.expand
    • lucid.chunk
    • lucid.masked_fill
    • lucid.roll
    • lucid.gather
    • lucid.unbind
    • lucid.sort
    • lucid.argsort
    • lucid.argmin
    • lucid.argmax
    • lucid.nonzero
    • lucid.unique
    • lucid.topk
    • lucid.histogram
    • lucid.histogram2d
    • lucid.histogramdd
    • lucid.where
    • lucid.diagonal

Autograd

  • lucid.autograd
  • Autograd APIs
    • lucid.autograd.backward
    • lucid.autograd.grad

Linear Algebra

  • lucid.linalg
  • Linalg Operations
    • lucid.linalg.det
    • lucid.linalg.inv
    • lucid.linalg.solve
    • lucid.linalg.norm
    • lucid.linalg.cholesky
    • lucid.linalg.eig
    • lucid.linalg.qr
    • lucid.linalg.svd
    • lucid.linalg.matrix_power
    • lucid.linalg.pinv

Random

  • lucid.random
  • RNG Functions
    • lucid.random.seed
    • lucid.random.rand
    • lucid.random.randint
    • lucid.random.randn
    • lucid.random.uniform
    • lucid.random.bernoulli
    • lucid.random.permutation

Einstein Operations

  • lucid.einops
  • Einops Functions
    • lucid.einops.rearrange
    • lucid.einops.reduce
    • lucid.einops.repeat
    • lucid.einops.einsum

Neural Networks

  • lucid.nn
  • nn.Parameter
  • nn.Buffer
  • nn.Module
  • Module Hooks
  • Neural Functions
    • Linear Functions
      • nn.functional.linear
      • nn.functional.bilinear
    • Activation Functions
      • nn.functional.relu
      • nn.functional.leaky_relu
      • nn.functional.elu
      • nn.functional.selu
      • nn.functional.gelu
      • nn.functional.sigmoid
      • nn.functional.tanh
      • nn.functional.softmax
    • Attention Functions
      • nn.functional.scaled_dot_product_attention
    • Convolution Functions
      • nn.functional.unfold
      • nn.functional.conv1d
      • nn.functional.conv2d
      • nn.functional.conv3d
      • nn.functional.conv_transpose1d
      • nn.functional.conv_transpose2d
      • nn.functional.conv_transpose3d
    • Pooling Functions
      • nn.functional.avg_pool1d
      • nn.functional.avg_pool2d
      • nn.functional.avg_pool3d
      • nn.functional.max_pool1d
      • nn.functional.max_pool2d
      • nn.functional.max_pool3d
      • nn.functional.adaptive_avg_pool1d
      • nn.functional.adaptive_avg_pool2d
      • nn.functional.adaptive_avg_pool3d
      • nn.functional.adaptive_max_pool1d
      • nn.functional.adaptive_max_pool2d
      • nn.functional.adaptive_max_pool3d
    • Dropout Functions
      • nn.functional.dropout
      • nn.functional.dropout1d
      • nn.functional.dropout2d
      • nn.functional.dropout3d
      • nn.functional.alpha_dropout
      • nn.functional.drop_block
      • nn.functional.drop_path
    • Normalization Functions
      • nn.functional.normalize
      • nn.functional.batch_norm
      • nn.functional.layer_norm
      • nn.functional.instance_norm
      • nn.functional.group_norm
      • nn.functional.global_response_norm
    • Loss Functions
      • nn.functional.mse_loss
      • nn.functional.binary_cross_entropy
      • nn.functional.binary_cross_entropy_with_logits
      • nn.functional.cross_entropy
      • nn.functional.nll_loss
      • nn.functional.huber_loss
    • Spatial Functions
      • nn.functional.affine_grid
      • nn.functional.grid_sample
    • Embedding Functions
      • nn.functional.embedding
      • nn.functional.sinusoidal_pos_embedding
      • nn.functional.rotary_pos_embedding
    • Utility Functions
      • nn.functional.interpolate
      • nn.functional.rotate
      • nn.functional.one_hot
  • Weight Initializations
    • nn.init.uniform
    • nn.init.normal
    • nn.init.constant
    • nn.init.xavier_uniform
    • nn.init.xavier_normal
    • nn.init.kaiming_uniform
    • nn.init.kaiming_normal
  • Modules
    • Linear Layers
      • nn.Identity
      • nn.Flatten
      • nn.Linear
      • nn.Bilinear
    • Convolution Layers
      • nn.Unfold
      • nn.Conv1d
      • nn.Conv2d
      • nn.Conv3d
      • nn.ConvTranspose1d
      • nn.ConvTranspose2d
      • nn.ConvTranspose3d
      • nn.ConstrainedConv1d
      • nn.ConstrainedConv2d
      • nn.ConstrainedConv3d
    • Activation Layers
      • nn.ReLU
      • nn.ReLU6
      • nn.LeakyReLU
      • nn.ELU
      • nn.SELU
      • nn.GELU
      • nn.Sigmoid
      • nn.HardSigmoid
      • nn.Tanh
      • nn.Softmax
      • nn.Swish
      • nn.HardSwish
      • nn.Mish
    • Pooling Layers
      • nn.AvgPool1d
      • nn.AvgPool2d
      • nn.AvgPool3d
      • nn.MaxPool1d
      • nn.MaxPool2d
      • nn.MaxPool3d
      • nn.AdaptiveAvgPool1d
      • nn.AdaptiveAvgPool2d
      • nn.AdaptiveAvgPool3d
      • nn.AdaptiveMaxPool1d
      • nn.AdaptiveMaxPool2d
      • nn.AdaptiveMaxPool3d
    • Normalization Layers
      • nn.BatchNorm1d
      • nn.BatchNorm2d
      • nn.BatchNorm3d
      • nn.InstanceNorm1d
      • nn.InstanceNorm2d
      • nn.InstanceNorm3d
      • nn.LayerNorm
      • nn.GroupNorm
      • nn.GlobalResponseNorm
    • Dropout Layers
      • nn.Dropout
      • nn.Dropout1d
      • nn.Dropout2d
      • nn.Dropout3d
      • nn.AlphaDropout
      • nn.DropBlock
      • nn.DropPath
    • Loss Layers
      • nn.MSELoss
      • nn.BCELoss
      • nn.BCEWithLogitsLoss
      • nn.CrossEntropyLoss
      • nn.NLLLoss
      • nn.HuberLoss
    • Vision Layers
      • nn.Upsample
    • Sparse Layers
      • nn.Embedding
    • Recurrent Layers
      • nn.RNNBase
      • nn.RNN
      • nn.LSTM
      • nn.GRU
      • nn.RNNCell
      • nn.LSTMCell
      • nn.GRUCell
    • Attention Layers
      • nn.ScaledDotProductAttention
      • nn.MultiHeadAttention
    • Transformer Layers
      • nn.TransformerEncoderLayer
      • nn.TransformerDecoderLayer
      • nn.TransformerEncoder
      • nn.TransformerDecoder
      • nn.Transformer
      • nn.SinusoidalPosEmbedding
      • nn.LearnedPosEmbedding
      • nn.RotaryPosEmbedding
    • Einops Layers
      • nn.Rearrange
  • Fused Modules
    • nn.ConvBNReLU1d
    • nn.ConvBNReLU2d
    • nn.ConvBNReLU3d
    • nn.DepthSeparableConv1d
    • nn.DepthSeparableConv2d
    • nn.DepthSeparableConv3d
    • nn.SEModule
    • nn.SelectiveKernel
  • Containers
    • nn.Sequential
    • nn.ModuleList
    • nn.ModuleDict
    • nn.ParameterList
    • nn.ParameterDict
  • Caches
    • nn.Cache
    • nn.KVCache
    • nn.EncoderDecoderCache
    • nn.DynamicKVCache
    • nn.StaticKVCache
  • Utilities
    • nn.utils.grad_norm
    • nn.utils.get_total_norm
    • nn.utils.clip_grad_norm
    • nn.utils.clip_grad_value
    • nn.utils.apply_chunking_to_forward
    • RNN Utilities
      • nn.utils.rnn.pad_sequence
      • nn.utils.rnn.PackedSequence
      • nn.utils.rnn.pack_padded_sequence
      • nn.utils.rnn.pad_packed_sequence
      • nn.utils.rnn.pack_sequence
      • nn.utils.rnn.unpack_sequence

Optimization

  • lucid.optim
  • optim.Optimizer
  • optim.lr_scheduler
  • Optimizers
    • optim.SGD
    • optim.ASGD
    • optim.RMSprop
    • optim.Rprop
    • optim.Adam
    • optim.AdamW
    • optim.NAdam
    • optim.RAdam
    • optim.Adamax
    • optim.Adagrad
    • optim.Adadelta
  • LR Schedulers
    • lr_scheduler.LRScheduler
    • lr_scheduler.LambdaLR
    • lr_scheduler.StepLR
    • lr_scheduler.MultiStepLR
    • lr_scheduler.ExponentialLR
    • lr_scheduler.CosineAnnealingLR
    • lr_scheduler.ReduceLROnPlateau
    • lr_scheduler.CyclicLR
    • lr_scheduler.NoamScheduler

Data

  • lucid.data
  • data.Dataset
  • data.Subset
  • data.TensorDataset
  • data.ConcatDataset
  • data.DataLoader
  • Tokenizers
    • tokenizers.Tokenizer
    • WordPieceTokenizer
    • WordPieceTokenizerFast
    • BPETokenizer
    • BPETokenizerFast
    • ByteBPETokenizer
    • ByteBPETokenizerFast
  • Utilities
    • data.random_split

Datasets

  • lucid.datasets
  • Image Datasets
    • MNIST
    • FashionMNIST
    • CIFAR10
    • CIFAR100

Models

  • lucid.models
  • Base Mixins
    • PreTrainedModelMixin
  • Vision Models
    • Image Classification
      • LeNet
        • LeNetConfig
        • lenet_1
        • lenet_4
        • lenet_5
      • AlexNet
        • AlexNetConfig
        • alexnet
      • ZFNet
        • ZFNetConfig
        • zfnet
      • VGGNet
        • VGGNetConfig
        • vggnet_11
        • vggnet_13
        • vggnet_16
        • vggnet_19
      • Inception
        • InceptionConfig
        • inception_v1
        • inception_v3
        • inception_v4
      • Inception-ResNet
        • InceptionResNetConfig
        • inception_resnet_v1
        • inception_resnet_v2
      • ResNet
        • ResNetConfig
        • resnet_18
        • resnet_34
        • resnet_50
        • resnet_101
        • resnet_152
        • resnet_200
        • resnet_269
        • resnet_1001
        • wide_resnet_50
        • wide_resnet_101
      • ResNeXt
        • ResNeXtConfig
        • resnext_50_32x4d
        • resnext_101_32x4d
        • resnext_101_32x8d
        • resnext_101_32x16d
        • resnext_101_32x32d
        • resnext_101_64x4d
      • ResNeSt
        • ResNeStConfig
        • resnest_14
        • resnest_26
        • resnest_50
        • resnest_101
        • resnest_200
        • resnest_269
        • resnest_50_4s2x40d
        • resnest_50_1s4x24d
      • SENet
        • SENetConfig
        • se_resnet_18
        • se_resnet_34
        • se_resnet_50
        • se_resnet_101
        • se_resnet_152
        • se_resnext_50_32x4d
        • se_resnext_101_32x4d
        • se_resnext_101_32x8d
        • se_resnext_101_64x4d
      • SKNet
        • SKNetConfig
        • sk_resnet_18
        • sk_resnet_34
        • sk_resnet_50
        • sk_resnext_50_32x4d
      • DenseNet
        • DenseNetConfig
        • densenet_121
        • densenet_169
        • densenet_201
        • densenet_264
      • Xception
        • XceptionConfig
        • xception
      • MobileNet
        • MobileNetConfig
        • mobilenet
      • MobileNet-v2
        • MobileNetV2Config
        • mobilenet_v2
      • MobileNet-v3
        • MobileNetV3Config
        • mobilenet_v3_small
        • mobilenet_v3_large
      • MobileNet-v4
        • MobileNetV4Config
        • mobilenet_v4_conv_small
        • mobilenet_v4_conv_medium
        • mobilenet_v4_conv_large
        • mobilenet_v4_hybrid_medium
        • mobilenet_v4_hybrid_large
      • EfficientNet
        • EfficientNetConfig
        • efficientnet_b0
        • efficientnet_b1
        • efficientnet_b2
        • efficientnet_b3
        • efficientnet_b4
        • efficientnet_b5
        • efficientnet_b6
        • efficientnet_b7
      • EfficientNet-v2
        • EfficientNetV2Config
        • efficientnet_v2_s
        • efficientnet_v2_m
        • efficientnet_v2_l
        • efficientnet_v2_xl
      • ConvNeXt
        • ConvNeXtConfig
        • convnext_tiny
        • convnext_small
        • convnext_base
        • convnext_large
        • convnext_xlarge
      • ConvNeXt-v2
        • ConvNeXtV2Config
        • convnext_v2_atto
        • convnext_v2_femto
        • convnext_v2_pico
        • convnext_v2_nano
        • convnext_v2_tiny
        • convnext_v2_base
        • convnext_v2_large
        • convnext_v2_huge
      • InceptionNeXt
        • InceptionNeXtConfig
        • inception_next_atto
        • inception_next_tiny
        • inception_next_small
        • inception_next_base
      • CoAtNet
        • CoAtNetConfig
        • coatnet_0
        • coatnet_1
        • coatnet_2
        • coatnet_3
        • coatnet_4
        • coatnet_5
        • coatnet_6
        • coatnet_7
      • CSPNet
        • CSPNetConfig
        • csp_resnet_50
        • csp_resnext_50_32x4d
        • csp_darknet_53
      • ViT
        • ViTConfig
        • vit_tiny
        • vit_small
        • vit_base
        • vit_large
        • vit_huge
      • Swin Transformer
        • SwinTransformerConfig
        • swin_tiny
        • swin_small
        • swin_base
        • swin_large
      • Swin Transformer-v2
        • SwinTransformerV2Config
        • swin_v2_tiny
        • swin_v2_small
        • swin_v2_base
        • swin_v2_large
        • swin_v2_huge
        • swin_v2_giant
      • CvT
        • CvTConfig
        • cvt_13
        • cvt_21
        • cvt_w24
      • PVT
        • PVTConfig
        • pvt_tiny
        • pvt_small
        • pvt_medium
        • pvt_large
        • pvt_huge
      • PVT-v2
        • PVTV2Config
        • pvt_v2_b0
        • pvt_v2_b1
        • pvt_v2_b2
        • pvt_v2_b2_li
        • pvt_v2_b3
        • pvt_v2_b4
        • pvt_v2_b5
      • CrossViT
        • CrossViTConfig
        • crossvit_tiny
        • crossvit_small
        • crossvit_base
        • crossvit_9
        • crossvit_15
        • crossvit_18
        • crossvit_9_dagger
        • crossvit_15_dagger
        • crossvit_18_dagger
      • MaxViT
        • MaxViTConfig
        • maxvit_tiny
        • maxvit_small
        • maxvit_base
        • maxvit_large
        • maxvit_xlarge
      • EfficientFormer
        • EfficientFormerConfig
        • efficientformer_l1
        • efficientformer_l3
        • efficientformer_l7
    • Object Detection
      • R-CNN
        • RCNNConfig
      • Fast R-CNN
        • FastRCNNConfig
      • Faster R-CNN
        • FasterRCNNConfig
        • faster_rcnn_resnet_50_fpn
        • faster_rcnn_resnet_101_fpn
      • YOLO
        • YOLO-v1
          • YOLO_V1Config
          • yolo_v1
          • yolo_v1_tiny
        • YOLO-v2
          • YOLO_V2Config
          • yolo_v2
          • yolo_v2_tiny
        • YOLO-v3
          • YOLO_V3Config
          • yolo_v3
          • yolo_v3_tiny
        • YOLO-v4
          • YOLO_V4Config
          • yolo_v4
      • EfficientDet
        • EfficientDetConfig
        • efficientdet_d0
        • efficientdet_d1
        • efficientdet_d2
        • efficientdet_d3
        • efficientdet_d4
        • efficientdet_d5
        • efficientdet_d6
        • efficientdet_d7
      • DETR
        • DETRConfig
        • detr_r50
        • detr_r101
    • Image Segmentation
      • FCN
        • FCNConfig
        • fcn_resnet_50
        • fcn_resnet_101
      • U-Net
        • UNetConfig
        • UNetStageConfig
        • U-Net 2D
        • U-Net 3D
      • ResUNet
        • ResUNet 2D
        • ResUNet 3D
      • Attention U-Net
        • AttentionUNetConfig
        • AttentionUNetGateConfig
        • Attention U-Net 2D
        • Attention U-Net 3D
      • Mask R-CNN
        • MaskRCNNConfig
        • mask_rcnn_resnet_50_fpn
        • mask_rcnn_resnet_101_fpn
      • MaskFormer
        • MaskFormerConfig
        • maskformer_resnet_18
        • maskformer_resnet_34
        • maskformer_resnet_50
        • maskformer_resnet_101
      • Mask2Former
        • Mask2FormerConfig
        • mask2former_resnet_18
        • mask2former_resnet_34
        • mask2former_resnet_50
        • mask2former_resnet_101
        • mask2former_swin_tiny
        • mask2former_swin_small
        • mask2former_swin_base
        • mask2former_swin_large
  • Generative Models
    • VAE
      • VAEConfig
    • DDPM
      • DDPMConfig
    • NCSN
      • NCSNConfig
  • Text Models
    • Transformer
      • TransformerConfig
      • transformer_base
      • transformer_big
    • BERT
      • BERTConfig
      • BERTTokenizerFast
      • BERTForPreTraining
      • BERTForMaskedLM
      • BERTForCausalLM
      • BERTForNextSentencePrediction
      • BERTForSequenceClassification
      • BERTForTokenClassification
      • BERTForQuestionAnswering
    • RoFormer
      • RoFormerConfig
      • RoFormerTokenizerFast
      • RoFormerForMaskedLM
      • RoFormerForSequenceClassification
      • RoFormerForTokenClassification
      • RoFormerForMultipleChoice
      • RoFormerForQuestionAnswering
    • GPT
      • GPTConfig
      • GPTLMHeadModel
      • GPTDoubleHeadsModel
      • GPTForSequenceClassification
  • Model Utilities
    • util.iou
    • util.nms
    • util.bbox_to_delta
    • util.apply_deltas
    • util.clip_boxes
    • util.SelectiveSearch
    • util.ROIAlign
    • util.MultiScaleROIAlign
    • util.FPN

Weights

  • lucid.weights
  • Pre-Trained Weights

Transformation

  • lucid.transforms
  • transforms.Compose
  • transforms.ToTensor
  • Image Transforms
    • transforms.Normalize
    • transforms.Resize
    • transforms.RandomHorizontalFlip
    • transforms.RandomVerticalFlip
    • transforms.RandomCrop
    • transforms.CenterCrop
    • transforms.RandomRotation
    • transforms.RandomGrayscale

Compilation

  • JIT Compilation
  • lucid.compile

Visualization

  • lucid.visual
  • Mermaid Charts
    • visual.build_tensor_mermaid_chart
    • visual.build_module_mermaid_chart

Porting

  • lucid.save
  • lucid.load

Others

  • lucid.Numeric
  • lucid.no_grad
  • lucid.grad_enabled
  • lucid.count_flops
  • lucid.newaxis
  • lucid.register_model
  1. Lucid /
  2. Modules /
  3. Attention Layers
View Source Open in ChatGPT Open in Claude

Attention LayersΒΆ

  • nn.ScaledDotProductAttention
  • nn.MultiHeadAttention
Previous
nn.GRUCell
Next
nn.ScaledDotProductAttention

2026, ChanLumerico

Made with Sphinx and Shibuya theme.