Skip to content

task_card

allennlp.common.task_card

[SOURCE]


A specification for defining task cards (derived from model cards). Motivation: A model's capabilities and limitations are dependent on the task definition. Thus, it is helpful to separate the information in the model card that comes from specifically the task itself.

ExampleCategory

@dataclass(frozen=True)
class ExampleCategory(FromParams)

category

class ExampleCategory(FromParams):
 | ...
 | category: str = None

examples

class ExampleCategory(FromParams):
 | ...
 | examples: List[Dict[str, str]] = None

TaskCard

@dataclass(frozen=True)
class TaskCard(FromParams)

The TaskCard stores information about the task. It is modeled after the ModelCard.

Parameters

  • id : str
    The task id. Example: "rc" for reading comprehension.
  • name : str, optional
    The (display) name of the task.
  • description : str, optional
    Description of the task. Example: "Textual Entailment (TE) is the task of predicting whether, for a pair of sentences, the facts in the first sentence necessarily imply the facts in the second."
  • expected_inputs : str, optional
    All expected inputs and their format. Example: (For a reading comprehension task) Passage (text string), Question (text string)
  • expected_outputs : str, optional
    All expected outputs and their format. Example: (For a reading comprehension task) Answer span (start token position and end token position).
  • examples : Union[List[Dict[str, str]], Dict[str, List[Dict[str, str]]]], optional
    List of examples for the task. Each example dict should contain as keys the expected_inputs. Example: (For textual entailment) [{"premise": "A handmade djembe was on display at the Smithsonian.", "hypothesis": "Visitors could see the djembe."}]
  • scope_and_limitations : str, optional
    This discusses the scope of the task based on how it is defined, and any limitations. Example: "The Textual Entailment task is in some sense "NLP-complete", and you should not expect any current model to cover every possible aspect of entailment. Instead, you should think about what the model was trained on to see whether it could reasonably capture the phenomena that you are querying it with."

id

class TaskCard(FromParams):
 | ...
 | id: str = None

name

class TaskCard(FromParams):
 | ...
 | name: Optional[str] = None

description

class TaskCard(FromParams):
 | ...
 | description: Optional[str] = None

expected_inputs

class TaskCard(FromParams):
 | ...
 | expected_inputs: Optional[str] = None

expected_outputs

class TaskCard(FromParams):
 | ...
 | expected_outputs: Optional[str] = None

scope_and_limitations

class TaskCard(FromParams):
 | ...
 | scope_and_limitations: Optional[str] = None

examples

class TaskCard(FromParams):
 | ...
 | examples: Optional[Union[List[Dict[str, str]], List[ExampleCategory]]] = None