Skip to content

find_learning_rate

allennlp.commands.find_learning_rate

[SOURCE]

The find-lr subcommand can be used to find a good learning rate for a model. It requires a configuration file and a directory in which to write the results.

FindLearningRate

@Subcommand.register("find-lr")
class FindLearningRate(Subcommand)

add_subparser

class FindLearningRate(Subcommand):
 | ...
 | def add_subparser(
 |     self,
 |     parser: argparse._SubParsersAction
 | ) -> argparse.ArgumentParser

find_learning_rate_from_args

def find_learning_rate_from_args(args: argparse.Namespace) -> None

Start learning rate finder for given args

find_learning_rate_model

def find_learning_rate_model(
    params: Params,
    serialization_dir: str,
    start_lr: float = 1e-5,
    end_lr: float = 10,
    num_batches: int = 100,
    linear_steps: bool = False,
    stopping_factor: float = None,
    force: bool = False
) -> None

Runs learning rate search for given num_batches and saves the results in serialization_dir

Parameters

  • params : Params
    A parameter object specifying an AllenNLP Experiment.
  • serialization_dir : str
    The directory in which to save results.
  • start_lr : float
    Learning rate to start the search.
  • end_lr : float
    Learning rate upto which search is done.
  • num_batches : int
    Number of mini-batches to run Learning rate finder.
  • linear_steps : bool
    Increase learning rate linearly if False exponentially.
  • stopping_factor : float
    Stop the search when the current loss exceeds the best loss recorded by multiple of stopping factor. If None search proceeds till the end_lr
  • force : bool
    If True and the serialization directory already exists, everything in it will be removed prior to finding the learning rate.

search_learning_rate

def search_learning_rate(
    trainer: GradientDescentTrainer,
    start_lr: float = 1e-5,
    end_lr: float = 10,
    num_batches: int = 100,
    linear_steps: bool = False,
    stopping_factor: float = None
) -> Tuple[List[float], List[float]]

Runs training loop on the model using GradientDescentTrainer increasing learning rate from start_lr to end_lr recording the losses.

Parameters

  • trainer : GradientDescentTrainer
  • start_lr : float
    The learning rate to start the search.
  • end_lr : float
    The learning rate upto which search is done.
  • num_batches : int
    Number of batches to run the learning rate finder.
  • linear_steps : bool
    Increase learning rate linearly if False exponentially.
  • stopping_factor : float
    Stop the search when the current loss exceeds the best loss recorded by multiple of stopping factor. If None search proceeds till the end_lr

Returns

  • (learning_rates, losses) : Tuple[List[float], List[float]]
    Returns list of learning rates and corresponding losses. Note: The losses are recorded before applying the corresponding learning rate