Skip to content

highway

[ allennlp.modules.highway ]

A Highway layer that does a gated combination of a linear transformation and a non-linear transformation of its input.

Highway#

class Highway(torch.nn.Module):
 | def __init__(
 |     self,
 |     input_dim: int,
 |     num_layers: int = 1,
 |     activation: Callable[[torch.Tensor], torch.Tensor] = torch.nn.functional.relu
 | ) -> None

A Highway layer does a gated combination of a linear transformation and a non-linear transformation of its input. :math:y = g * x + (1 - g) * f(A(x)), where :math:A is a linear transformation, :math:f is an element-wise non-linearity, and :math:g is an element-wise gate, computed as :math:sigmoid(B(x)).

This module will apply a fixed number of highway layers to its input, returning the final result.

Parameters

  • input_dim : int
    The dimensionality of :math:x. We assume the input has shape (batch_size, ..., input_dim).
  • num_layers : int, optional (default = 1)
    The number of highway layers to apply to the input.
  • activation : Callable[[torch.Tensor], torch.Tensor], optional (default = torch.nn.functional.relu)
    The non-linearity to use in the highway layers.

forward#

class Highway(torch.nn.Module):
 | ...
 | @overrides
 | def forward(self, inputs: torch.Tensor) -> torch.Tensor