find_learning_rate

allennlp.commands.find_learning_rate

[SOURCE]

---

The find-lr subcommand can be used to find a good learning rate for a model. It requires a configuration file and a directory in which to write the results.

FindLearningRate

@Subcommand.register("find-lr")
class FindLearningRate(Subcommand)

add_subparser

class FindLearningRate(Subcommand):
 | ...
 | @overrides
 | def add_subparser(
 |     self,
 |     parser: argparse._SubParsersAction
 | ) -> argparse.ArgumentParser

find_learning_rate_from_args

def find_learning_rate_from_args(args: argparse.Namespace) -> None

Start learning rate finder for given args

find_learning_rate_model

def find_learning_rate_model(
    params: Params,
    serialization_dir: str,
    start_lr: float = 1e-5,
    end_lr: float = 10,
    num_batches: int = 100,
    linear_steps: bool = False,
    stopping_factor: float = None,
    force: bool = False
) -> None

Runs learning rate search for given num_batches and saves the results in serialization_dir

Parameters

• params : Params
  A parameter object specifying an AllenNLP Experiment.
• serialization_dir : str
  The directory in which to save results.
• start_lr : float
  Learning rate to start the search.
• end_lr : float
  Learning rate upto which search is done.
• num_batches : int
  Number of mini-batches to run Learning rate finder.
• linear_steps : bool
  Increase learning rate linearly if False exponentially.
• stopping_factor : float
  Stop the search when the current loss exceeds the best loss recorded by multiple of stopping factor. If None search proceeds till the end_lr
• force : bool
  If True and the serialization directory already exists, everything in it will be removed prior to finding the learning rate.

search_learning_rate

def search_learning_rate(
    trainer: GradientDescentTrainer,
    start_lr: float = 1e-5,
    end_lr: float = 10,
    num_batches: int = 100,
    linear_steps: bool = False,
    stopping_factor: float = None
) -> Tuple[List[float], List[float]]

Runs training loop on the model using GradientDescentTrainer increasing learning rate from start_lr to end_lr recording the losses.

Parameters

• trainer : GradientDescentTrainer
  
• start_lr : float
  The learning rate to start the search.
• end_lr : float
  The learning rate upto which search is done.
• num_batches : int
  Number of batches to run the learning rate finder.
• linear_steps : bool
  Increase learning rate linearly if False exponentially.
• stopping_factor : float
  Stop the search when the current loss exceeds the best loss recorded by multiple of stopping factor. If None search proceeds till the end_lr

Returns

• (learning_rates, losses) : Tuple[List[float], List[float]]
  Returns list of learning rates and corresponding losses. Note: The losses are recorded before applying the corresponding learning rate