Skip to content

linear_attention

allennlp.modules.attention.linear_attention

[SOURCE]

LinearAttention

@Attention.register("linear")
class LinearAttention(Attention):
 | def __init__(
 |     self,
 |     tensor_1_dim: int,
 |     tensor_2_dim: int,
 |     combination: str = "x,y",
 |     activation: Activation = None,
 |     normalize: bool = True
 | ) -> None

This Attention module performs a dot product between a vector of weights and some combination of the two input vectors, followed by an (optional) activation function. The combination used is configurable.

If the two vectors are x and y, we allow the following kinds of combinations : x, y, x*y, x+y, x-y, x/y, where each of those binary operations is performed elementwise. You can list as many combinations as you want, comma separated. For example, you might give x,y,x*y as the combination parameter to this class. The computed similarity function would then be w^T [x; y; x*y] + b, where w is a vector of weights, b is a bias parameter, and [;] is vector concatenation.

Note that if you want a bilinear similarity function with a diagonal weight matrix W, where the similarity function is computed as x * w * y + b (with w the diagonal of W), you can accomplish that with this class by using "x*y" for combination.

Registered as an Attention with name "linear".

Parameters

  • tensor_1_dim : int
    The dimension of the first tensor, x, described above. This is x.size()[-1] - the length of the vector that will go into the similarity computation. We need this so we can build weight vectors correctly.
  • tensor_2_dim : int
    The dimension of the second tensor, y, described above. This is y.size()[-1] - the length of the vector that will go into the similarity computation. We need this so we can build weight vectors correctly.
  • combination : str, optional (default = "x,y")
    Described above.
  • activation : Activation, optional (default = linear)
    An activation function applied after the w^T * [x;y] + b calculation. Default is linear, i.e. no activation.
  • normalize : bool, optional (default = True)

reset_parameters

class LinearAttention(Attention):
 | ...
 | def reset_parameters(self)