allennlp.data.dataset_readers.masked_language_modeling¶

class allennlp.data.dataset_readers.masked_language_modeling.MaskedLanguageModelingReader(tokenizer: allennlp.data.tokenizers.tokenizer.Tokenizer = None, token_indexers: Dict[str, allennlp.data.token_indexers.token_indexer.TokenIndexer] = None, lazy: bool = False)[source]¶

Bases: allennlp.data.dataset_readers.dataset_reader.DatasetReader

Reads a text file and converts it into a Dataset suitable for training a masked language model.

The Field s that we create are the following: an input TextField, a mask position ListField[IndexField], and a target token TextField (the target tokens aren’t a single string of text, but we use a TextField so we can index the target tokens the same way as our input, typically with a single PretrainedTransformerIndexer). The mask position and target token lists are the same length.

NOTE: This is not fully functional! It was written to put together a demo for interpreting and attacking masked language modeling, not for actually training anything. text_to_instance is functional, but _read is not. To make this fully functional, you would want some sampling strategies for picking the locations for [MASK] tokens, and probably a bunch of efficiency / multi-processing stuff.

Parameters

tokenizerTokenizer, optional (default=``WordTokenizer()``): We use this Tokenizer for the text. See Tokenizer.
token_indexersDict[str, TokenIndexer], optional (default=``{“tokens”: SingleIdTokenIndexer()}``): We use this to define the input representation for the text, and to get ids for the mask targets. See TokenIndexer.

text_to_instance(self, sentence: str = None, tokens: List[allennlp.data.tokenizers.token.Token] = None, targets: List[str] = None) → allennlp.data.instance.Instance[source]¶

Parameters

sentencestr, optional: A sentence containing [MASK] tokens that should be filled in by the model. This input is superceded and ignored if tokens is given.
tokensList[Token], optional: An already-tokenized sentence containing some number of [MASK] tokens to be predicted.
targetsList[str], optional: Contains the target tokens to be predicted. The length of this list should be the same as the number of [MASK] tokens in the input.