allennlp.data.dataset_readers.text_classification_json

class allennlp.data.dataset_readers.text_classification_json.TextClassificationJsonReader(token_indexers: Dict[str, allennlp.data.token_indexers.token_indexer.TokenIndexer] = None, tokenizer: allennlp.data.tokenizers.tokenizer.Tokenizer = None, segment_sentences: bool = False, max_sequence_length: int = None, skip_label_indexing: bool = False, lazy: bool = False)

Bases: allennlp.data.dataset_readers.dataset_reader.DatasetReader

Reads tokens and their labels from a labeled text classification dataset. Expects a “text” field and a “label” field in JSON format.

The output of read is a list of Instance s with the fields:

tokens: TextField and label: LabelField

Parameters

token_indexersDict[str, TokenIndexer], optional

optional (default=``{“tokens”: SingleIdTokenIndexer()}``) We use this to define the input representation for the text. See TokenIndexer.

tokenizerTokenizer, optional (default = {"tokens": WordTokenizer()})

Tokenizer to use to split the input text into words or other kinds of tokens.

segment_sentences: ``bool``, optional (default = ``False``)

If True, we will first segment the text into sentences using SpaCy and then tokenize words. Necessary for some models that require pre-segmentation of sentences, like the Hierarchical Attention Network (https://www.cs.cmu.edu/~hovy/papers/16HLT-hierarchical-attention-networks.pdf).

max_sequence_length: ``int``, optional (default = ``None``)

If specified, will truncate tokens to specified maximum length.

skip_label_indexing: ``bool``, optional (default = ``False``)

Whether or not to skip label indexing. You might want to skip label indexing if your labels are numbers, so the dataset reader doesn’t re-number them starting from 0.

lazybool, optional, (default = False)

Whether or not instances can be read lazily.

text_to_instance(self, text: str, label: Union[str, int] = None) → allennlp.data.instance.Instance

Parameters

textstr, required.

The text to classify

labelstr, optional, (default = None).

The label for this text.

Returns

An Instance containing the following fields:

tokensTextField

The tokens in the sentence or phrase.

labelLabelField

The label label of the sentence or phrase.