Skip to content

model_test_case

allennlp.common.testing.model_test_case

[SOURCE]

ModelTestCase

class ModelTestCase(AllenNlpTestCase)

A subclass of AllenNlpTestCase with added methods for testing Model subclasses.

set_up_model

class ModelTestCase(AllenNlpTestCase):
 | ...
 | def set_up_model(
 |     self,
 |     param_file: PathLike,
 |     dataset_file: PathLike,
 |     serialization_dir: PathLike = None,
 |     seed: int = None
 | )

test_model_batch_norm_verification

class ModelTestCase(AllenNlpTestCase):
 | ...
 | def test_model_batch_norm_verification(self)

ensure_model_can_train_save_and_load

class ModelTestCase(AllenNlpTestCase):
 | ...
 | def ensure_model_can_train_save_and_load(
 |     self,
 |     param_file: Union[PathLike, str],
 |     tolerance: float = 1e-4,
 |     cuda_device: int = -1,
 |     gradients_to_ignore: Set[str] = None,
 |     overrides: str = "",
 |     metric_to_check: str = None,
 |     metric_terminal_value: float = None,
 |     metric_tolerance: float = 1e-4,
 |     disable_dropout: bool = True,
 |     which_loss: str = "loss",
 |     seed: int = None
 | )

Parameters

  • param_file : str
    Path to a training configuration file that we will use to train the model for this test.
  • tolerance : float, optional (default = 1e-4)
    When comparing model predictions between the originally-trained model and the model after saving and loading, we will use this tolerance value (passed as rtol to numpy.testing.assert_allclose).
  • cuda_device : int, optional (default = -1)
    The device to run the test on.
  • gradients_to_ignore : Set[str], optional (default = None)
    This test runs a gradient check to make sure that we're actually computing gradients for all of the parameters in the model. If you really want to ignore certain parameters when doing that check, you can pass their names here. This is not recommended unless you're really sure you don't need to have non-zero gradients for those parameters (e.g., some of the beam search / state machine models have infrequently-used parameters that are hard to force the model to use in a small test).
  • overrides : str, optional (default = "")
    A JSON string that we will use to override values in the input parameter file.
  • metric_to_check : str, optional (default = None)
    We may want to automatically perform a check that model reaches given metric when training (on validation set, if it is specified). It may be useful in CI, for example. You can pass any metric that is in your model returned metrics.
  • metric_terminal_value : str, optional (default = None)
    When you set metric_to_check, you need to set the value this metric must converge to
  • metric_tolerance : float, optional (default = 1e-4)
    Tolerance to check you model metric against metric terminal value. One can expect some variance in model metrics when the training process is highly stochastic.
  • disable_dropout : bool, optional (default = True)
    If True we will set all dropout to 0 before checking gradients. (Otherwise, with small datasets, you may get zero gradients because of unlucky dropout.)
  • which_loss : str, optional (default = "loss")
    Specifies which loss to test. For example, which_loss may be "adversary_loss" for adversarial_bias_mitigator.

ensure_model_can_train

class ModelTestCase(AllenNlpTestCase):
 | ...
 | def ensure_model_can_train(
 |     self,
 |     trainer: GradientDescentTrainer,
 |     gradients_to_ignore: Set[str] = None,
 |     metric_to_check: str = None,
 |     metric_terminal_value: float = None,
 |     metric_tolerance: float = 1e-4,
 |     disable_dropout: bool = True
 | )

A simple test for model training behavior when you are not using configuration files. In this case, we don't have a story around saving and loading models (you need to handle that yourself), so we don't have tests for that. We just test that the model can train, and that it computes gradients for all parameters.

Because the Trainer already has a reference to a model and to a data loader, we just take the Trainer object itself, and grab the Model and other necessary objects from there.

Parameters

  • trainer : GradientDescentTrainer
    The Trainer to use for the test, which already has references to a Model and a DataLoader, which we will use in the test.
  • gradients_to_ignore : Set[str], optional (default = None)
    This test runs a gradient check to make sure that we're actually computing gradients for all of the parameters in the model. If you really want to ignore certain parameters when doing that check, you can pass their names here. This is not recommended unless you're really sure you don't need to have non-zero gradients for those parameters (e.g., some of the beam search / state machine models have infrequently-used parameters that are hard to force the model to use in a small test).
  • metric_to_check : str, optional (default = None)
    We may want to automatically perform a check that model reaches given metric when training (on validation set, if it is specified). It may be useful in CI, for example. You can pass any metric that is in your model returned metrics.
  • metric_terminal_value : str, optional (default = None)
    When you set metric_to_check, you need to set the value this metric must converge to
  • metric_tolerance : float, optional (default = 1e-4)
    Tolerance to check you model metric against metric terminal value. One can expect some variance in model metrics when the training process is highly stochastic.
  • disable_dropout : bool, optional (default = True)
    If True we will set all dropout to 0 before checking gradients. (Otherwise, with small datasets, you may get zero gradients because of unlucky dropout.)

assert_fields_equal

class ModelTestCase(AllenNlpTestCase):
 | ...
 | def assert_fields_equal(
 |     self,
 |     field1,
 |     field2,
 |     name: str,
 |     tolerance: float = 1e-6
 | ) -> None

check_model_computes_gradients_correctly

class ModelTestCase(AllenNlpTestCase):
 | ...
 | @staticmethod
 | def check_model_computes_gradients_correctly(
 |     model: Model,
 |     model_batch: Dict[str, Union[Any, Dict[str, Any]]],
 |     params_to_ignore: Set[str] = None,
 |     disable_dropout: bool = True,
 |     which_loss: str = "loss"
 | )

ensure_batch_predictions_are_consistent

class ModelTestCase(AllenNlpTestCase):
 | ...
 | def ensure_batch_predictions_are_consistent(
 |     self,
 |     keys_to_ignore: Iterable[str] = ()
 | )

Ensures that the model performs the same on a batch of instances as on individual instances. Ignores metrics matching the regexp .loss. and those specified explicitly.

Parameters

  • keys_to_ignore : Iterable[str], optional (default = ())
    Names of metrics that should not be taken into account, e.g. "batch_weight".