mindformers.core.WarmUpStableDecayLR

View Source On Gitee
class mindformers.core.WarmUpStableDecayLR(learning_rate: float, lr_end: float = 1e-7, warmup_steps: int = None, warmup_lr_init: float = 0., warmup_ratio: float = None, total_steps: int = None, decay_start_steps: int = None, decay_start_ratio: float = None, **kwargs)[source]

Warm Up Stable Decay Learning Rate.

Parameters

  • learning_rate (float) – Initial value of learning rate.

  • lr_end (float, optional) – Final value of learning rate. Default: 0..

  • warmup_steps (int, optional) – The number of warm up steps. Default: None.

  • warmup_lr_init (float, optional) – Initial learning rate in warm up steps. Default: 0..

  • warmup_ratio (float, optional) – Ratio of total training steps used for warmup. Default: None.

  • total_steps (int, optional) – The number of total steps. Default: None.

  • decay_start_steps (int, optional) – The start step of decay. Default: None.

  • decay_start_ratio (float, optional) – Ratio of total training steps used for decay. Default: None.

Raises

- ValueError - If lr_end is greater than or equal to initial learning_rate. –

Inputs:

  • global_step (int) - The global step.

Outputs:

Learning rate.