Skip to content

conll2003

allennlp.data.dataset_readers.conll2003

[SOURCE]

Conll2003DatasetReader

@DatasetReader.register("conll2003")
class Conll2003DatasetReader(DatasetReader):
 | def __init__(
 |     self,
 |     token_indexers: Dict[str, TokenIndexer] = None,
 |     tag_label: str = "ner",
 |     feature_labels: Sequence[str] = (),
 |     convert_to_coding_scheme: Optional[str] = None,
 |     label_namespace: str = "labels",
 |     **kwargs
 | ) -> None

Reads instances from a pretokenised file where each line is in the following format:

WORD POS-TAG CHUNK-TAG NER-TAG

with a blank line indicating the end of each sentence and -DOCSTART- -X- -X- O indicating the end of each article, and converts it into a Dataset suitable for sequence tagging.

Each Instance contains the words in the "tokens" TextField. The values corresponding to the tag_label values will get loaded into the "tags" SequenceLabelField. And if you specify any feature_labels (you probably shouldn't), the corresponding values will get loaded into their own SequenceLabelField s.

This dataset reader ignores the "article" divisions and simply treats each sentence as an independent Instance. (Technically the reader splits sentences on any combination of blank lines and "DOCSTART" tags; in particular, it does the right thing on well formed inputs.)

Registered as a DatasetReader with name "conll2003".

Parameters

  • token_indexers : Dict[str, TokenIndexer], optional (default = {"tokens": SingleIdTokenIndexer()})
    We use this to define the input representation for the text. See TokenIndexer.
  • tag_label : str, optional (default = ner)
    Specify ner, pos, or chunk to have that tag loaded into the instance field tag.
  • feature_labels : Sequence[str], optional (default = ())
    These labels will be loaded as features into the corresponding instance fields: pos -> pos_tags, chunk -> chunk_tags, ner -> ner_tags Each will have its own namespace : pos_tags, chunk_tags, ner_tags. If you want to use one of the tags as a feature in your model, it should be specified here.
  • convert_to_coding_scheme : str, optional (default = None)
    Specifies the coding scheme for ner_labels and chunk_labels. Conll2003DatasetReader assumes a coding scheme of input data is IOB1. Valid options are None and BIOUL. The None default maintains the original IOB1 scheme in the CoNLL 2003 NER data. In the IOB1 scheme, I is a token inside a span, O is a token outside a span and B is the beginning of span immediately following another span of the same type.
  • coding_scheme : str, optional (default = IOB1)
    This parameter is deprecated. If you specify coding_scheme to IOB1, consider simply removing it or specifying convert_to_coding_scheme to None. If you want to specify BIOUL for coding_scheme, replace it with convert_to_coding_scheme.
  • label_namespace : str, optional (default = labels)
    Specifies the namespace for the chosen tag_label.

text_to_instance

class Conll2003DatasetReader(DatasetReader):
 | ...
 | def text_to_instance(
 |     self,
 |     tokens: List[Token],
 |     pos_tags: List[str] = None,
 |     chunk_tags: List[str] = None,
 |     ner_tags: List[str] = None
 | ) -> Instance

We take pre-tokenized input here, because we don't have a tokenizer in this class.

apply_token_indexers

class Conll2003DatasetReader(DatasetReader):
 | ...
 | def apply_token_indexers(self, instance: Instance) -> None