allennlp.modules.elmo_lstm

A stacked bidirectional LSTM with skip connections between layers.

class allennlp.modules.elmo_lstm.ElmoLstm(input_size: int, hidden_size: int, cell_size: int, num_layers: int, requires_grad: bool = False, recurrent_dropout_probability: float = 0.0, memory_cell_clip_value: Optional[float] = None, state_projection_clip_value: Optional[float] = None)

Bases: allennlp.modules.encoder_base._EncoderBase

A stacked, bidirectional LSTM which uses LstmCellWithProjection’s with highway layers between the inputs to layers. The inputs to the forward and backward directions are independent - forward and backward states are not concatenated between layers.

Additionally, this LSTM maintains its own state, which is updated every time forward is called. It is dynamically resized for different batch sizes and is designed for use with non-continuous inputs (i.e inputs which aren’t formatted as a stream, such as text used for a language modeling task, which is how stateful RNNs are typically used). This is non-standard, but can be thought of as having an “end of sentence” state, which is carried across different sentences.

Parameters

input_sizeint, required

The dimension of the inputs to the LSTM.

hidden_sizeint, required

The dimension of the outputs of the LSTM.

cell_sizeint, required.

The dimension of the memory cell of the LstmCellWithProjection.

num_layersint, required

The number of bidirectional LSTMs to use.

requires_grad: ``bool``, optional

If True, compute gradient of ELMo parameters for fine tuning.

recurrent_dropout_probability: ``float``, optional (default = 0.0)

The dropout probability to be used in a dropout scheme as stated in A Theoretically Grounded Application of Dropout in Recurrent Neural Networks .

state_projection_clip_value: ``float``, optional, (default = None)

The magnitude with which to clip the hidden_state after projecting it.

memory_cell_clip_value: ``float``, optional, (default = None)

The magnitude with which to clip the memory cell.

forward(self, inputs: torch.Tensor, mask: torch.LongTensor) → torch.Tensor

Parameters

inputstorch.Tensor, required.

A Tensor of shape (batch_size, sequence_length, hidden_size).

masktorch.LongTensor, required.

A binary mask of shape (batch_size, sequence_length) representing the non-padded elements in each sequence in the batch.

Returns

A torch.Tensor of shape (num_layers, batch_size, sequence_length, hidden_size),

where the num_layers dimension represents the LSTM output from that layer.

load_weights(self, weight_file: str) → None

Load the pre-trained weights from the file.