bidirectional_lm
BidirectionalLanguageModel#
class BidirectionalLanguageModel(LanguageModel):
| def __init__(
| self,
| vocab: Vocabulary,
| text_field_embedder: TextFieldEmbedder,
| contextualizer: Seq2SeqEncoder,
| dropout: float = None,
| num_samples: int = None,
| sparse_embeddings: bool = False,
| initializer: InitializerApplicator = None,
| **kwargs
| ) -> None
The BidirectionalLanguageModel
applies a bidirectional "contextualizing"
Seq2SeqEncoder
to uncontextualized embeddings, using a SoftmaxLoss
module (defined above) to compute the language modeling loss.
It is IMPORTANT that your bidirectional Seq2SeqEncoder
does not do any
"peeking ahead". That is, for its forward direction it should only consider
embeddings at previous timesteps, and for its backward direction only embeddings
at subsequent timesteps. If this condition is not met, your language model is
cheating.
Parameters
- vocab :
Vocabulary
- text_field_embedder :
TextFieldEmbedder
Used to embed the indexed tokens we get inforward
. - contextualizer :
Seq2SeqEncoder
Used to "contextualize" the embeddings. As described above, this encoder must not cheat by peeking ahead. - dropout :
float
, optional (default =None
)
If specified, dropout is applied to the contextualized embeddings before computation of the softmax. The contextualized embeddings themselves are returned without dropout. - num_samples :
int
, optional (default =None
)
If provided, the model will useSampledSoftmaxLoss
with the specified number of samples. Otherwise, it will use the full_SoftmaxLoss
defined above. - sparse_embeddings :
bool
, optional (default =False
)
Passed on toSampledSoftmaxLoss
if True.