Skip to content

transformer_layer

allennlp.modules.transformer.transformer_layer

[SOURCE]

AttentionLayer

class AttentionLayer(TransformerModule,  FromParams):
 | def __init__(
 |     self,
 |     hidden_size: int,
 |     num_attention_heads: int,
 |     attention_dropout: float = 0.0,
 |     hidden_dropout: float = 0.0
 | )

This module wraps the self-attention with the output-layer, similar to the architecture in BERT. Details in the paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al, 2019

Parameters

  • hidden_size : int
  • num_attention_heads : int
  • attention_dropout : float, optional (default = 0.0)
    Dropout probability for the SelfAttention layer.
  • hidden_dropout : float, optional (default = 0.0)
    Dropout probability for the OutputLayer.

forward

class AttentionLayer(TransformerModule,  FromParams):
 | ...
 | def forward(
 |     self,
 |     input_tensor: torch.Tensor,
 |     attention_mask: torch.BoolTensor,
 |     head_mask: Optional[torch.Tensor] = None,
 |     encoder_hidden_states: Optional[torch.Tensor] = None,
 |     encoder_attention_mask: Optional[torch.BoolTensor] = None,
 |     output_attentions: bool = False
 | )

input_tensor : torch.Tensor Shape batch_size x seq_len x hidden_dim attention_mask : torch.BoolTensor, optional Shape batch_size x seq_len head_mask : torch.BoolTensor, optional output_attentions : bool Whether to also return the attention probabilities, default = False

TransformerLayer

class TransformerLayer(TransformerModule,  FromParams):
 | def __init__(
 |     self,
 |     hidden_size: int,
 |     intermediate_size: int,
 |     num_attention_heads: int,
 |     attention_dropout: float = 0.0,
 |     hidden_dropout: float = 0.0,
 |     activation: Union[str, torch.nn.Module] = "relu",
 |     add_cross_attention: bool = False
 | )

This module is a single transformer layer, mapping to BertLayer in the architecture in BERT. Details in the paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al, 2019

Parameters

  • hidden_size : int
  • intermediate_size : int
  • num_attention_heads : int
  • attention_dropout : float, optional (default = 0.0)
    Dropout probability for the SelfAttention layer.
  • hidden_dropout : float, optional (default = 0.0)
    Dropout probability for the OutputLayer.
  • activation : Union[str, torch.nn.Module]
  • add_cross_attention : bool, optional (default = False)
    If True, an extra AttentionLayer is added for cross-attention. This is helpful when using the layer in a decoder.

forward

class TransformerLayer(TransformerModule,  FromParams):
 | ...
 | def forward(
 |     self,
 |     hidden_states: torch.Tensor,
 |     attention_mask: torch.Tensor,
 |     head_mask: Optional[torch.Tensor] = None,
 |     encoder_hidden_states: Optional[torch.Tensor] = None,
 |     encoder_attention_mask: Optional[torch.Tensor] = None,
 |     output_attentions: bool = False
 | )

hidden_states : torch.Tensor Shape batch_size x seq_len x hidden_dim attention_mask : torch.BoolTensor, optional Shape batch_size x seq_len head_mask : torch.BoolTensor, optional encoder_hidden_states : torch.Tensor, optional encoder_attention_mask : torch.Tensor, optional output_attentions : bool Whether to also return the attention probabilities, default = False