RoFormerForMaskedLM¶

class lucid.models.RoFormerForMaskedLM(config: RoFormerConfig)¶

The RoFormerForMaskedLM class attaches a masked language modeling head to the RoFormer backbone.

Class Signature¶

class RoFormerForMaskedLM(config: RoFormerConfig)

Parameters¶

config (RoFormerConfig): RoFormer configuration for masked language modeling.

Methods¶

RoFormerForMaskedLM.forward(input_ids: LongTensor | None = None, attention_mask: Tensor | None = None, token_type_ids: LongTensor | None = None, position_ids: LongTensor | None = None, inputs_embeds: FloatTensor | None = None) → Tensor

RoFormerForMaskedLM.get_loss(labels: Tensor, input_ids: LongTensor | None = None, attention_mask: Tensor | None = None, token_type_ids: LongTensor | None = None, position_ids: LongTensor | None = None, inputs_embeds: FloatTensor | None = None, *, ignore_index: int = -100, reduction: str | None = 'mean') → Tensor

RoFormerForMaskedLM.create_masked_lm_inputs(input_ids: Tensor, attention_mask: Tensor | None = None, special_tokens_mask: Tensor | None = None, *, mask_token_id: int = 103, mlm_probability: float = 0.15, mask_replace_prob: float = 0.8, random_replace_prob: float = 0.1, ignore_index: int = -100) → tuple[Tensor, Tensor]

RoFormerForMaskedLM.predict_token_ids(input_ids: LongTensor | None = None, attention_mask: Tensor | None = None, token_type_ids: LongTensor | None = None, position_ids: LongTensor | None = None, inputs_embeds: FloatTensor | None = None) → Tensor

RoFormerForMaskedLM.get_accuracy(labels: Tensor, input_ids: LongTensor | None = None, attention_mask: Tensor | None = None, token_type_ids: LongTensor | None = None, position_ids: LongTensor | None = None, inputs_embeds: FloatTensor | None = None, *, ignore_index: int = -100) → Tensor

RoFormerForMaskedLM.get_loss_from_text(tokenizer: BERTTokenizerFast, text_a: str, text_b: str | None = None, *, device: Literal['cpu', 'gpu'] = 'cpu', mask_token_id: int | None = None, mlm_probability: float = 0.15, mask_replace_prob: float = 0.8, random_replace_prob: float = 0.1, ignore_index: int = -100, reduction: str | None = 'mean') → Tensor

RoFormerForMaskedLM.predict_token_ids_from_text(tokenizer: BERTTokenizerFast, text_a: str, text_b: str | None = None, *, device: Literal['cpu', 'gpu'] = 'cpu') → Tensor

Examples¶

>>> import lucid.models as models
>>> config = models.RoFormerConfig.base(vocab_size=50000)
>>> model = models.RoFormerForMaskedLM(config)
>>> print(model)
RoFormerForMaskedLM(...)