Skip to content

drop

allennlp_models.rc.tools.drop

[SOURCE]

EXCLUDE#

EXCLUDE = set(string.punctuation)

get_metrics#

def get_metrics(
    predicted: Union[str, List[str], Tuple[str, ...]],
    gold: Union[str, List[str], Tuple[str, ...]]
) -> Tuple[float, float]

Takes a predicted answer and a gold answer (that are both either a string or a list of strings), and returns exact match and the DROP F1 metric for the prediction. If you are writing a script for evaluating objects in memory (say, the output of predictions during validation, or while training), this is the function you want to call, after using answer_json_to_strings when reading the gold answer from the released data file.

answer_json_to_strings#

def answer_json_to_strings(
    answer: Dict[str, Any]
) -> Tuple[Tuple[str, ...], str]

Takes an answer JSON blob from the DROP data release and converts it into strings used for evaluation.

evaluate_json#

def evaluate_json(
    annotations: Dict[str, Any],
    predicted_answers: Dict[str, Any]
) -> Tuple[float, float]

Takes gold annotations and predicted answers and evaluates the predictions for each question in the gold annotations. Both JSON dictionaries must have query_id keys, which are used to match predictions to gold annotations (note that these are somewhat deep in the JSON for the gold annotations, but must be top-level keys in the predicted answers).

The annotations are assumed to have the format of the dev set in the DROP data release. The predicted_answers JSON must be a dictionary keyed by query id, where the value is a string (or list of strings) that is the answer.

evaluate_prediction_file#

def evaluate_prediction_file(
    prediction_path: str,
    gold_path: str,
    output_path: Optional[str] = None
) -> Tuple[float, float]

Takes a prediction file and a gold file and evaluates the predictions for each question in the gold file. Both files must be json formatted and must have query_id keys, which are used to match predictions to gold annotations. The gold file is assumed to have the format of the dev set in the DROP data release. The prediction file must be a JSON dictionary keyed by query id, where the value is either a JSON dictionary with an "answer" key, or just a string (or list of strings) that is the answer. Writes a json with global_em and global_f1 metrics to file at the specified output path, unless None is passed as output path.